The Relationship Between Teacher Education

THE RELATIONSHIP BETWEEN TEACHER EDUCATION COURSES AND THE TECHNOLOGY INTEGRATION ATTITUDES, BELIEFS, AND KNOWLEDGE OF PRESERVICE TEACHERS: A SYSTEMATIC REVIEW AND META-ANALYSIS SERIES

MATTHEW L. WILSON

A DISSERTATION PRESENTED TO THE GRADUATE SCHOOL OF THE UNIVERSITY OF FLORIDA IN PARTIAL FULFILLMENT OF THE REQUIREMENTS FOR THE DEGREE OF DOCTOR OF PHILOSOPHY

UNIVERSITY OF FLORIDA

2018

To my loving wife, Jungsun. This process would not have happened without you.

ACKNOWLEDGMENTS

There are many people to thank, but first I want to thank God. Through faith all things are possible. Without the strength and guidance of God’s grace, I don’t believe I could have endured to the end. It’s been a long journey from repatriating and moving across the country, to starting school and teaching at the University of Florida, to bring my wife to the U.S.A., and it goes on and on. The unending love of God helped me get through it all, and now I am here.

I acknowledge the immense contribution of Albert Ritzhaupt. Without his post to ISTE for doctoral studies at UF, I would never have thought to come start this amazing process.

Through his guidance and support, I have conducted award winning research, achieved more than I would have expected academically, and grown as a person and academic. His guidance will always be appreciated.

I acknowledge the support of my committee. Kara Dawson has been a wonderful supporter of me both academically and professionally. I appreciate all the work we have done together, and I look forward to the future. Nicholas Gage has helped shepherd me through this process of systematic review and meta-analysis. Without the frequent visits to his office, I am sure this work would never have gotten done. Finally, Anne Corinne Huggins-Manley has been a superb mentor and colleague in research methodology. Once again, I look forward to future collaborations with her.

I would be remiss if I did not mention the contributions Li Cheng made to this research. I thank her for stepping in to help with the coding process. Her help improved the quality of this research, and could not have been completed without her.

I would like to acknowledge my peers. There were times when this whole process was incredibly daunting. Knowing that others were going through the same process and having the same struggles somehow made it all seem possible.

I would like to thank my family. My father, Jon, who pride in my accomplishments was unmeasured. I miss you, Pop. I hope you are looking on this, and beaming. To my mother,

Rosemary, who always supports my choices. I know you too are proud of this, too. Finally, to my sister, Molly, who is a doctor in her own rights. You made it through, and so did I.

Finally, a special thanks to my wife, Jungsun. Moving from Korea to Gainesville was not easy for us. It was quite the departure from where we were. Yet, I am now finished up, and we can move forward to a new and interesting next step. I love you very much, and I am glad you were here with me.

TABLE OF CONTENTS

page

ACKNOWLEDGMENTS ...... 4

LIST OF TABLES ...... 10

LIST OF FIGURES ...... 13

ABSTRACT ...... 14

CHAPTER

1 INTRODUCTION ...... 16

Background ...... 16 Problem Statement ...... 18 Research Questions ...... 19 Purpose of Study ...... 20 Research Design ...... 22 Definition of Key Study Terms ...... 24 Organization of the Dissertation ...... 25

2 LITERATURE REVIEW ...... 26

Technology Integration in PK12 ...... 26 A Historical Perspective ...... 27 History of Technology Use for Education ...... 27 History of Technology Integration Focused Teacher Education ...... 30 Defining and Illustrating PK12 Technology Integration ...... 33 PK12 Technology Integration: A Definition ...... 33 PK12 Technology Integration in Practice ...... 37 Barriers to Technology Integration in PK12 Classrooms ...... 46 Examination of the Outcome Variables ...... 55 Understanding Attitudes, Beliefs, and Knowledge ...... 56 Attitudes ...... 56 Beliefs ...... 64 Knowledge ...... 68 Measurement of Attitudes, Beliefs, and Knowledge ...... 75 Changes to Second-Order Barriers Through Education ...... 82 Changes in attitudes ...... 82 Changes in beliefs ...... 83 Changes in knowledge ...... 84 Identified Features of Teacher Education Courses for Technology Integration ...... 85 Mentoring/Coaching ...... 86 Rehearsal/Field-Experience ...... 87 Goal-Setting ...... 88

Observation ...... 88 Reflection/Self-Evaluation ...... 89 Hands-On Learning ...... 90 Work Sample Analysis ...... 90 Practice Lesson Planning ...... 91 Conclusion ...... 91

3 METHODS ...... 92

Introduction ...... 92 Meta-Analysis Overview ...... 94 Recommendations for Systematic Review and Meta-Analysis ...... 97 Step 1: Formulating the Problem ...... 98 Step 2: Searching the Literature ...... 99 Step 3: Gathering Information from Studies ...... 101 Effect size calculations ...... 103 Step 4: Evaluating Study Quality ...... 105 Step 5: Analyzing and Integrating the Outcome of Studies ...... 106 Main effect estimation ...... 106 Sub-group analyses ...... 107 Heterogeneity analysis ...... 108 Step 6: Interpreting the Evidence ...... 108 Step 7: Presenting the Results ...... 109 Study Methods ...... 110 Step 1: Formulating the Problem ...... 111 Step 2: Searching the Literature ...... 111 Step 3: Gathering Information from Studies ...... 113 Step 4: Evaluating the Quality of Studies ...... 120 Step 5: Analyzing and Integrating the Outcomes of Studies ...... 121 Effect size estimation and aggregation ...... 121 Step 6: Interpreting the Evidence ...... 123 Main effect estimation ...... 123 Sub-group analyses ...... 124 Heterogeneity analysis ...... 126 Step 7: Presenting the Results ...... 127 Conclusion ...... 127

4 RESULTS ...... 128

Systematic Review ...... 128 Description of the Studies ...... 129 Study Publication Description ...... 129 Study Population and Context Descriptions ...... 130 Methodological and Measurement Tool Descriptions ...... 132 Publication Bias ...... 135 Trim-and-Fill Plots ...... 135 Fail-Safe N Results ...... 138

Heterogeneity Estimation ...... 139 Outcome Variable Results ...... 140 Attitude ...... 141 Beliefs ...... 142 Knowledge ...... 143 Sub-group Analyses ...... 144 Course Features ...... 144 Study Quality Impacts ...... 149 Study quality rank ...... 149 Measurement Validity ...... 151 Reported Reliability ...... 152 Conclusion ...... 153

5 DISCUSSION ...... 154

Introduction ...... 154 Main Research Question ...... 155 Attitudes ...... 155 Beliefs ...... 158 Knowledge ...... 160 Implications of Main Variable Outcomes ...... 163 Subquestion Interpretations and Implications ...... 165 Subquestion One - Course Features ...... 165 Subquestion Two - Study Quality ...... 167 Subquestion Three - Validity ...... 167 Subquestion Four - Reported Reliability ...... 168 Heterogeneity ...... 169 Limitations and Delimitations ...... 170 Future Research ...... 177 Conclusion ...... 179

APPENDIX

A INDEPENDENT SAMPLE DESIGN FORMULAS AND TERMINOLOGY TABLES ...183

B APA META-ANALYSIS CHECKLIST ...... 184

C PRISMA CHECKLIST ...... 186

D TITLE AND ABSTRACT CODING PROTOCOL ...... 188

E SECOND ROUND CODING PROTOCOL ...... 191

F DRAFT FULL CODING PROTOCOL...... 197

G FULL CODING PROTOCOL ...... 204

H FINALIZED CODEBOOK ...... 217

I PRISMA FLOWCHART...... 223

J DATA EXAMPLE ...... 224

K R CODE MARKDOWN DOCUMENT - ATTITUDE...... 225

L R CODE MARKDOWN DOCUMENT - BELIEFS...... 249

M R CODE MARKDOWN DOCUMENT - KNOWLEDGE ...... 271

N ATTITUDE STUDIES ...... 295

O BELIEFS STUDIES ...... 298

P KNOWLEDGE STUDIES ...... 299

Q ATTITUDE RESEARCH DESIGN DATA ...... 303

R BELIEFS RESEARCH DESIGN DATA ...... 310

S KNOWLEDGE RESEARCH DESIGN DATA ...... 317

LIST OF REFERENCES ...... 325

BIOGRAPHICAL SKETCH ...... 342

LIST OF TABLES

Table page

1-1 Key study term definitions...... 24

2-1 Descriptors from ISTE Standards for Students 2016 from International Society for Technology in Education...... 41

2-2 Descriptions of ISTE’s Essential Conditions...... 48

2-3 A summary list of the barriers affecting technology integration in PK12 classrooms...... 51

2-4 Examples of technological knowledge...... 75

2-5 Examples of technology integration attitudes, beliefs, or knowledge measures...... 77

3-1 Outline of the meta-analytic process from Cooper...... 97

3-2 Search criteria...... 112

3-3 Preliminary criterion for study quality ...... 120

4-1 Attitudes studies...... 129

4-2 Beliefs studies...... 129

4-3 Knowledge studies...... 129

4-4 Course credit hours of technology integration courses in primary sources...... 130

4-5 Course level of technology integration courses in primary sources...... 131

4-6 Study location of technology integration course university in primary sources...... 131

4-7 Study design of the primary source studies...... 133

4-8 Self-reported measures in primary source studies...... 133

4-9 Calculated Fail-safe Ns...... 139

4-10 I2 estimates of heterogeneity in outcome variable measures...... 140

4-11 Comparison of attitude outcome by mentoring/coaching...... 145

4-12 Comparison of attitude outcome by field experience/rehearsal...... 145

4-13 Comparison of attitude outcome by goal-setting...... 145

4-14 Comparison of attitude outcome by observation...... 145

4-15 Comparison of attitude outcome by reflection/self-evaluation...... 145

4-16 Comparison of attitude outcome by hands-on learning...... 145

4-17 Comparison of attitude outcome by work sample analysis...... 145

4-18 Comparison of attitude outcome by lesson planning...... 146

4-19 Comparison of beliefs outcome by mentoring/coaching...... 146

4-20 Comparison of beliefs outcome by field experience/rehearsal...... 146

4-21 Comparison of beliefs outcome by observation...... 147

4-22 Comparison of beliefs outcome by reflection/self-evaluation...... 147

4-23 Comparison of beliefs outcome by hands-on learning...... 147

4-24 Comparison of beliefs outcome by work sample analysis...... 147

4-25 Comparison of beliefs outcome by lesson planning...... 147

4-26 Comparison of knowledge outcome by mentoring/coaching...... 148

4-27 Comparison of knowledge outcome by field experience/rehearsal...... 148

4-28 Comparison of knowledge outcome by goal-setting...... 148

4-29 Comparison of knowledge outcome by observation...... 148

4-30 Comparison of knowledge outcome by reflection/self-evaluation...... 148

4-31 Comparison of knowledge outcome by hands-on learning...... 148

4-32 Comparison of knowledge outcome by work sample analysis...... 148

4-33 Comparison of knowledge outcome by lesson planning...... 149

4-34 Comparison of attitude outcome by study quality rank...... 150

4-35 Comparison of beliefs outcome by study quality rank...... 150

4-36 Comparison of knowledge outcome by study quality rank...... 150

4-37 Comparison of attitude outcome by measurement validity level...... 151

4-38 Comparison of beliefs outcome by measurement validity level...... 151

4-39 Comparison of knowledge outcome by measurement validity level...... 151

4-40 Comparison of attitude outcome by reliability...... 152

4-41 Comparison of beliefs outcome by reliability...... 152

4-42 Comparison of knowledge outcome by reliability...... 152

A-1 Independent sample design formulas...... 183

A-2 Terminology...... 183

LIST OF FIGURES

page

2-1 Dale’s Cone of Experience...... 28

2-2 “Framework for 21st Century Learning”...... 36

2-3 Technology Integration Matrix Progression across levels of integration...... 42

2-4 Levels of the digital divide...... 45

2-5 Theorized relationships between technology integration barriers...... 48

2-6 Aligning Hew and Brush barriers with Ertmer barrier degrees...... 52

2-7 Visualization of framework guiding current meta-analysis...... 55

2-8 Attitudinal dimensions...... 57

2-9 Technology attitudes...... 61

2-10 TPACK framework...... 69

2-11 TPK continua...... 72

2-12 Findings on TPACK measurements from 2006-2010 ...... 81

3-1 Conceptualization of meta-analytic research...... 99

3-2 Sub-group analysis flowchart...... 107

4-1 Attitude variable trim-and-fill ...... 136

4-2 Belief variable trim-and-fill...... 137

4-3 Knowledge variable trim-and-fill...... 138

4-4 Attitude outcome forest plot...... 141

4-5 Belief outcome forest plot...... 142

4-6 Knowledge outcome forest plot...... 143

Abstract of Dissertation Presented to the Graduate School of the University of Florida in Partial Fulfillment of the Requirements for the Degree of Doctor of Philosophy

THE RELATIONSHIP BETWEEN TEACHER EDUCATION COURSES AND THE TECHNOLOGY INTEGRATION ATTITUDES, BELIEFS, AND KNOWLEDGE OF PRESERVICE TEACHERS: A SYSTEMATIC REVIEW AND META-ANALYSIS SERIES

Matthew L. Wilson

May 2018

Chair: Albert Ritzhaupt Major: Curriculum and Instruction

The integration of technology into PK12 classrooms is difficult for even experienced teachers. Research has shown there are multiple barriers to the integration of technology by teachers, including the attitudes, beliefs, and knowledge (i.e., second-order barriers; Ertmer,

1999) of those teachers. Researchers have been working for years to better understand how to prepare preservice teachers to teach at the highest level using technology to aid in the process of teaching and learning. This study sought to better understand how stand-alone technology integration courses influenced these second-order barriers.

This study utilized a systematic review and meta-analyses to investigate what, if any, effect teacher education courses for technology integration (TECTI) had on the attitudes, beliefs, and knowledge related to technology integration of preservice teachers. A systematic review of the literature was conducted from six databases focused on technology integration and/or teacher education. From an initial pool of 2,936 research studies, 55 studies were identified addressing one or more of these variables. Main effect meta-analyses were conducted using a random- effects model. A series of sub-group analyses were conducted to further investigate the influence

of course features (e.g., work sample analyses, practice lesson planning, etc.), study quality, measure validity, and reported reliability had on the estimation of the mean effect size.

The results of the study showed that TECTI had statistically significant results on the technology integration attitudes, beliefs, and knowledge of preservice teachers. In the sub-group analyses, none of the eight course features were determined to have a significant effect. Out of the three main variables, study quality was determined to have improved effect size estimation in attitude alone. Finally, neither measure validity or reported reliability had a statistically significant impact on effect size estimation. Further discussion, implications, and recommendations are explored.

CHAPTER 1 INTRODUCTION

Background

Technology can empower educators as engineers of collaboration on content, designers of learning experiences, leaders, guides, and catalysts of change; so that side-by-side with students new learning experiences for deeper exploration of content are achieved (U.S.

Department of Education (Office of Educational Technology), 2016). Teachers are increasingly expected to be effective facilitators, engaging in joint problem-solving with their students, but if teachers are unable to implement emerging digital tools in creative ways, this cannot be accomplished (Johnson, Adams-Becker, Estrada, & Freeman, 2015). Educational reform efforts, based on a new definition of “good” teaching through which teaching should facilitate student learning by leveraging relevant information and communications technology resources

(henceforth known as ICT) as meaningful pedagogical tools, have consistently purported student-centered practices as the most effective way to prepare our students for the 21st century

(Ertmer & Ottenbreit-Leftwich, 2013). Teacher education addresses technology integration through multiple methods, including: a single or stand-alone course or series of courses; mini- workshops; embedded integration into methods courses; and other program options (Kay, 2006).

In a study by Gronseth et al. (2010), they found that 60% of the institutions that responded to their survey required stand-alone educational technology courses for initial licensure of teachers.

To better understand how to integrate technology effectively within the classroom, it becomes increasingly important for academics and practitioners within the intersecting domains of teacher education and educational technology to understand what best practice looks like. Even with a strong body of literature related to technology integration education for preservice teachers and the resulting course design, individuals designing such courses may be unaware of this literature

and/or the influence of teacher education on the factors that prevent teachers from choosing to integrate technology into teaching and learning. These barriers are described in the literature as first-order barriers (e.g., resources, funding, etc.), and second-order barriers (e.g., attitudes, beliefs, and knowledge) (Ertmer, 1999). Considering the prevalence of teacher education courses for technology integration in teacher education programs, understanding the relationship between such courses and second-order barriers through an examination of empirical studies across the breadth of academic research becomes important and practical.

Technology integration in the PK12 classrooms takes on a variety of forms. There are potentially thousands of points of view for what it means to integrate technology into education, but some current frameworks and standards prevail. TPACK (formally TPCK) is a framework addressing the integration of technology into teaching. Conceptualized by Mishra and Koehler in

2006, Technological Pedagogical Content Knowledge (TPACK) is a widely used framework within teacher education and professional development, as well as educational research. Mishra and Koehler (2006) view TPACK as the intersection of three unique domains of knowledge

(technological, pedagogical, and content knowledge), the hybrid knowledge areas resulting from the intersection of the aforementioned domains, and preeminent intersection now known as

TPACK. The application of this hybrid knowledge allows for teacher to produce and implement technology integrated lessons. To work towards student goals, the International Society for

Technology in Education (ISTE) defines a series of standard sets for students. To teach in an integrated way, teachers should guide students toward sound digital citizenship, creative design and creation of digital artifacts, and global collaboration for the empowerment of learning

(International Society for Technology in Education, 2017h). Binkley et al. (2012) define this as ways of thinking and working with technology tools for living in the world, or 21st Century

skills. Thus, technology integration in PK12 context may be readily defined as the application of teacher knowledge towards the teaching and learning of 21st Century skills.

Problem Statement

The integration of technology into PK12 classrooms is difficult for even experienced teachers. Research shows there are multiple barriers to the integration of technology by teachers.

Ertmer (1999) explains that most teachers today are quick to recognize the importance technology in classrooms, barriers such as personal fears, technical and logistical issues, organizational and pedagogical concerns can block implementation efforts, and although teachers may not face all of these barriers any one of these barriers alone can significantly impede meaningful classroom use. Hew and Brush (2007) tentatively identify assessment, subject culture, teacher attitudes and beliefs, knowledge and skills, the institution (i.e., school culture), and resources as playing a role in technology integration. While some of these features are simply out of the control of teachers; beliefs, attitudes, and knowledge are components that must be central to teacher education courses on technology integration.

The beliefs, attitudes, knowledge of preservice educators, and the resulting instructional behaviors, developed through teacher education courses devoted to technology integration require close examination to better design such courses for maximum effect. Hew and Brush

(2007) argue that change in attitudes and beliefs related to technology integration requires four considerations: teachers’ knowledge and skills, subject culture, assessment, and institution support. Subject culture, assessment, and institution support are all potential extrinsic (i.e., first- order) barriers to technology integration by teachers (Ertmer, 1999). Conversely, attitudes and beliefs, as well as the resulting teaching behaviors, are intrinsic and can interfere with or impede fundamental change (Ertmer, 1999). Since such change results from a shift in attitudinal behavior, along a continuum from simple awareness and acceptance, technology integration

courses should looks at their impact for creating an affective shift in students (Morrison, Ross,

Kalman, & Kemp, 2013). To fail to prepare future teachers with the abilities, strategies, and ways of thinking for teaching today and tomorrow using technology will rob the children of tomorrow (Niess, 2008).

Research Questions

This study applied a systematic review process and meta-analytic methods to a series of evaluations into the magnitude of effect of teacher education courses for technology integration

(TECTI) on the attitudes, beliefs, and knowledge (i.e., second-order barriers constructs) of preservice teachers for technology integration in PK12 classrooms. The courses under examination are those in teacher education at universities with four- or five-year programs and stand-alone technology integration courses. These meta-analyses sought to analyze the relationship between teacher preparation courses and the development and shift of factors that act as barriers to technology use by preservice teachers in their future classrooms. Furthermore, the sub-group meta-analyses sought to analyze the impact moderating features of course feature, overall study quality, measurement validity, and reported reliability. Specifically, meta-analysis was applied to explore the following primary question of inquiry: What is the effect of TECTI on second-order barrier constructs (i.e., teacher attitudes, beliefs, and knowledge)? The subsequent subquestions were:

 What is the difference in magnitude of effect of TECTI on each second-order barrier construct when a course feature (i.e., mentoring, observation, rehearsal/field-experience, mentoring/coaching, etc.) is present or absent in a TECTI offering?

 What is the difference in magnitude of effect of TECTI on each second-order barrier construct when comparing levels of study quality?

 What is the difference in magnitude of effect of TECTI on each second-order barrier when comparing levels of measure validity?

 What is the difference in magnitude of effect of TECTI on each second-order barrier when comparing levels of reported reliability?

Purpose of Study

This research study has multiple goals in understanding the impact of teacher education courses for technology integration (TECTI) on the second-order barriers that cause teachers to avoid or resist technology integration in practice. Niess (2008) encourages researchers and teacher educators to inquire as to what experiences and how should the experiences be arranged so preservice teachers may develop the appropriate skills and knowledge given they lack experiences in guiding student learning both with and about technologies. This inquiry must also be extended to attitudes and beliefs, as those components have also been shown to impact the integration of technology by teachers. As such, the primary goal and question of this study seeks to quantify the relationship between technology integration courses and the beliefs, attitudes, and knowledge of preservice teachers related to technology integration. Having a clear understanding of the direction and magnitude of the potential shift in attitudes, beliefs, and knowledge provides a foundational guide for ongoing research into teacher education in this area. Subsequently, providing such a foundation potentially leads to early intervention in teacher practice related to technology integration. Ginsburg (2009) defines four stages of teacher development: apprenticeship of observation, preservice, induction, and inservice. Education courses devoted to technology integration provide opportunities for teacher educators to influence the direction of technology integration in PK12 classrooms. Since technology integration has been shown to be influenced by beliefs, attitudes, and knowledge in teachers (An & Reigeluth, 2011; Hew &

Brush, 2007; Inan & Lowther, 2010; Liu, Ritzhaupt, Dawson, & Barron, 2016; Reid, 2014;

Ritzhaupt, Dawson, & Cavanaugh, 2012), eliminating or mitigating such barriers early in the teacher development cycle could aid in the effective integration of technology PK12. Therefore,

a multi-study examination of the relationship between technology integration courses and the attitudes, beliefs, and knowledge of preservice teachers through a systematic review of the literature and a series of meta-analyses can clarify the research through a statistical summarizing of existing studies. With this information as a guide, researchers can expand and refine future research offers in this area. The targeted courses under examination are those in teacher education at universities with four- or five-year programs and stand-alone courses in technology integration.

The first subquestion, related to course features, examines the potential impact of specific course features in relation to change in the second-order barrier variables. Using the primary question results as a guide, exploring the specific impact of those features can provide multiple benefits to studies and practice in this domain. For researchers, it can provide added knowledge for continuing studies on teacher education, technology integration, course design, and impact on attitudes, beliefs, and knowledge related to those domains. For policy makers, it helps refine the nature of course programs and offerings related to technology integration and teacher education.

The final three subquestions (i.e., those questions on study quality, validity, and reliability) help clarify the results of TECTI impacts on second-order barriers. Each of these subgroup analyses further quantify what, if any, impact the study quality, validity, and reported reliability have on the estimation of average effect size of each second-order barrier. Such knowledge allows for researchers to better understand both what the true effect size of across studies may be, and thereby better interpret current research and conceptualize future studies.

As research is a “cooperative and cumulative enterprise,” an examination of past research is a necessary step in the development of knowledge (Cooper, 2017, p. 2). Furthermore, the ever- increasing call for evidence-based decision making means that an analysis of prior research can

provide a cumulative look at best practice in research and research results (Cooper, 2017). Meta- analysis allows for across study examination of consistency or variability in research, and provides a statistical conclusion in a way that narrative reviews cannot (Borenstein, Hedges,

Higgins, & Rothstein, 2009). As such, a systematic review of the literature and corresponding meta-analysis into both main effects and sub-groups can support the development of knowledge and understanding of best practice.

Research Design

In order to gain insight into the impact of TECTI, a series of meta-analyses were conceptualized and enacted. The primary analyses were conducted to investigate the impact of stand-alone technology courses on one of three second-order barriers to technology integration by teachers. The first of these barriers was teacher attitudes about technology and technology integration. For this study, attitudes were considered as affective positions by an individual related to technology or technology integration. These attitudes could have manifested as computer (technology) anxiety (Heinssen, Glass, & Knight, 1987; Meuter, Ostrom, Bitner, &

Roundtree, 2003), technophobia (Campion, 1989; Rosen & Weil, 1995b), technophilia

(Campion, 1989), computer (technology) confidence/self-efficacy (Lee & Lee, 2014; L. Wang,

Ertmer, & Newby, 2004), technology acceptance (Davis, 1993; Davis, Bagozzi, & Warshaw,

1989; Venkatesh, Morris, Davis, & Davis, 2003), technology adoption behaviors (Lin, 2003;

Parente & Prescott, 1994), perceived usefulness of technology (Davis, 1993), and technostress

(Ennis, 2005; Ragu-Nathan, Tarafdar, Ragu-Nathan, & Tu, 2008; K. Wang, Shu, & Tu, 2008).

The second barrier, beliefs, were those evaluative positions a teacher takes as to the value of technology for teaching and learning. Ertmer (2005) named these beliefs teacher pedagogical beliefs. Finally, knowledge was the integrated whole knowledge the teacher has related to technology and technology integration. Knowledge was comprised of both practical knowledge

of technology (i.e., ICT or digital literacy), and conceptual knowledge (i.e., technological pedagogical, technological content, and technological pedagogical content knowledge (Mishra &

Koehler, 2006)).

Using Cooper’s (2017) framework for systematic review and meta-analysis, studies related to the topic of this study were identified through a systematic review of teacher education literature related to technology integration courses, and a subsequent series of meta-analyses, both main effect and sub-group analyses, were conducted. Research literature was reviewed to identify studies in which the effect of TECTI were empirically measured in at least one of the three outcome variables in this study. Further sub-group and moderator analyses were conducted into the potential impact on effect size of course features, study quality, and measurement validity, and reliability. A review of the literature yielded eight course features that could potentially support changes in technology integration attitudes, beliefs, and knowledge: mentoring/coaching, rehearsal/field experience, goal-setting, observation, reflection/self- evaluation, hands-on learning, work sample analysis, and practice lesson planning. As such, this research considered the impact on the outcome variable if any of these features were present in the course design. While strict eligibility criteria were defined for the study, not all studies are designed equally. Quality judgments may be made a posteriori using an empirical evaluation of study quality (Cooper, 2017). However, these a posteriori evaluations may allow for the inclusion of poor quality studies to start (Cooper, 2017). Therefore, the impact of the study quality was evaluated to aid in the interpretation of the meta-analysis results. Finally, it was hypothesized that the primary studies identified for this research may use measures to assess the change in second-order barriers for which the validity of the tool or reliability of the data collected by said tool may not have been established. Therefore, assessment of validity and

reliability were incorporated into the study, and analyzed to determine how those features may have impacted the estimated average effect size across studies.

A random-effects model of the standardized mean difference was calculated using restricted maximum likelihood (REML) estimator for the analysis of each second-order barrier, as well as all sub-group analyses. Borenstein et al. (2009) explain that the random-effects model is best applied when the effect sizes under consideration are expected to be similar, but not identical, across studies because of variation in the participants. Additionally, the intervention under consideration in this study (i.e., TECTI) were most likely implemented using a variety of conditions, which will not be uniform. Therefore, the random-effects model best allowed for the approximation of the effects in all cases (Borenstein et al., 2009).

Definition of Key Study Terms

Table 1-1. Key study term definitions. Term Definition attitude The affective positions by an individual related to technology or technology integration. barrier The conditions both internal and external that discourage technology integration by teachers (Ertmer, 1999). belief The evaluative positions a teacher takes as to the value of technology for teaching and learning, named teacher pedagogical beliefs (Ertmer, 2005). first-order barrier Those barriers which are obstacles to technology integration extrinsic to teachers, for example missing or inadequate resources in teachers' implementation environments (Ertmer, 1999) knowledge Knowledge was comprised of both practical knowledge of technology (i.e., ICT or digital literacy), and the conceptual knowledge (i.e., technological pedagogical, technological content, and technological pedagogical and content knowledge (TPACK) (Mishra & Koehler, 2006)). preservice teacher Any student enrolled in a four- or five-year university program devoted to teacher education. second-order barrier Those barriers to technology integration that interfere with or impede fundamental change, and are rooted in teachers' underlying beliefs about teaching and learning, and may not be apparent to others or even to the teachers themselves (e.g., attitudes, beliefs, and knowledge) (Ertmer, 1999). teacher education course Any course related to teacher education as part of a university program. technological content An understanding of the manner in which technology and content influence knowledge (tck) and constrain one another (Koehler & Mishra, 2009).

Table 1-1. Continued. Term Definition technological knowledge Technological knowledge is certain ways of thinking about and working (tk) with technology can apply to all technology tools and resources (Koehler & Mishra, 2009). The skills related to the practical operation of technology. May also be called digital literacy, information literacy, technological literacy, etc. technological pedagogical An understanding that emerges from interactions among content, pedagogy, and content knowledge and technology knowledge (Koehler & Mishra, 2009). (TPACK) technological pedagogical An understanding of how teaching and learning can change when particular knowledge (tpk) technologies are used in particular ways (Koehler & Mishra, 2009). technology All digital technologies (e.g., computers, handheld devices, software applications etc.) that are protean (i.e., usable in many different ways), rapidly changing, and opaque (i.e., the inner workings are hidden from users) (Koehler & Mishra, 2009). technology integration The application of technology for teaching and learning in a classroom in any grade level PK12.

Organization of the Dissertation

This meta-analytic study is organized into five chapters. Chapter 1 defines the research problem and rationale for the research. Chapter 2 addresses the relevant literature on technology integration, barriers to technology integration by teachers, the outcome variables, and TECTI course features related to barrier change. Chapter 3 details an explanation of the methods of systematic review and meta-analysis in seven stages, and how those processes were applied for this current study. Chapter 4 presents the results of all related analyses. Chapter 5 includes a discussion of the findings and implications of the results, including: main effect and sub-group analyses results, interpretation, and implications; recommendations for future research; and limitations and delimitations of the study.

CHAPTER 2 LITERATURE REVIEW

The purpose of this study was to explore via meta-analysis how teacher education courses on technology integration impact the attitudes, beliefs and knowledge of preservice teachers

(PSTs) toward technology integration. Following here is a review of relevant literature related to this study. First, an overview of the history, practice, and policies of prekindergarten to Grade 12 technology integration is provided. The second section specifies an in-depth examination of the barriers to technology integration by teachers. The final section defines the factors under exploration through the meta-analytic research of this study, measures related to those factors, and how teacher education research recommends addressing technology integration.

Technology Integration in PK12

Technology integration in classrooms from prekindergarten to Grade 12 (PK12) can take on various forms and functions depending on the teacher in the classroom (Ritzhaupt et al.,

2012). Moreover, integrating technology is also a function of available resources, which could be mobile or desktop, Windows or Apple, or even the newest release or something multiple years old. The Association for Educational Communications and Technology (AECT; 2008) aptly explain that any conception of educational technology, and by extension technology integration, is constantly evolving and can only ever be a temporary one, a “snapshot in time” (p. 1). To frame technology integration for the study at hand, it is valuable to examine the history of technology use for education, define technology integration, provide a view of what technology integration looks as a practice in the context of PK12, and understand how each of these impacts how preservice teacher education.

A Historical Perspective

To better understand the nature of technology integration, it helps to understand how the practice has changed over time. The next two sections provide a situating look at two topics. The first is an overall history of technology integration. The second is a look at how policy and practice have influenced teacher education.

History of Technology Use for Education

While educational technology has shifted drastically over the course of time, exploring the nature of the change helps provide a clearer picture of educational technology today. As far back as Cro-Magnon times, primitive children were taught to observe, imitate, and participate in activities vital to the survival of the tribe (Saettler, 1990). Fast-forwarding to today, the technology integrated Common Core State Standards (CCSS) aim to prepare all students for success in our global economy and society (National Governors Association Center for Best

Practices, 2010). How did education, and the related educational technology, move from its beginnings to now? This section explores key elements of the evolution of technology use in education.

Saettler (1990) harks the history of modern educational technology back to earliest Elder

Sophists of ancient Greece (circa 500 BCE). One of the Sophists’ major tenets was that “Man evolves through technology and social organization to a state of civilization where he can guide his affairs effectively” (Saettler, 1990, p. 24). While the Sophists’ view of technology was the traditional meaning of the study of art or craft, a modern slant could be put on the view to connect it with a view of technology integration today. Saettler (1990) goes on to connect educational technology through the scholastic method, Comenius’ principles, Lancastrian methods, the through Pestalozzi, Froebel, and Herbart. While not fully comprehensive, Saettler

(1990) provides a set of concepts, selected from historical instructional theory and method, that could be considered precursors to modern educational technology.

Moving through the 1990s, various technological advances (i.e., devices and media) impacted the growth of technology in education. The earliest decades of the 20th century saw museums, film, and radio each play a role in the education (Reiser, 2001). Such media lead to the development by Dale of the Cone of Experience, based on the principle that all teaching can be

“greatly improved by visual and auditory materials because these teaching materials can make the learning experience far more concrete and memorable” (Dale, 1946, p. 6). The Cone of

Experience ranges from a base of direct, purposeful experiences (i.e., rich, full-bodied experiences which are seen, handled, tasted, felt, touches, and smelt) to verbal symbols, which might be best thought of as full abstractions (see Figure 2-1) (Dale, 1946). As will be shown later in this chapter, each of Dale’s ten experiences, may both now be experienced with and through technology.

Figure 2-1. Dale’s Cone of Experience. Adapted from Dale (1946).

The next stage of educational technology evolution involved a shift from media to technologies, as computer based/aided instruction gained momentum. Reiser (2001) explains that terminology moved away from media to the technology, both in academic organization and government focus. This trend has continued as computers have continued to shrink, mobilize, and become ubiquitous through the creation of the Internet.

As computerized technologies evolved, so did the view of their place in education. By the early 1980s, the value of technology for education was under serious scrutiny. In Clark’s 1983 article, Reconsidering Research on Learning from Media, he writes that summaries and meta- analyses of media comparison studies suggest media are merely vehicles that deliver instruction, but do not influence student achievement under any conditions. Furthermore, Clark (1983) contends that media research is vulnerable to rival hypotheses, citing evidence of artifact and confounding in existing studies, and bias in research publication selection. In the 1991 article,

Learning With Media, Kozma responds to Clark’s call for a moratorium on media research.

Kozma (1991) asserts that media is not merely a delivery device, but learners actively collaborate with the medium to construct knowledge through the cognitive resources of the medium and prior knowledge. Kozma (1991) clearly states that “our ability to take advantage of the power of emerging technologies will depend on the creativity of designers, their ability to exploit the capabilities of the media, and our understanding of the relationship between these capabilities and learning” (p. 206). This debate will likely go on as technology is further integrated into education. As more recent commentary shows, Selwyn (2015) argues that technology and education remains an area of academic study, policymaking, commercial activity and popular debate where promises of what might/could/should happen far outstrip the realities of what actually happens. Additionally, Tamim, Bernard, Borokhovski, Abrami, and Schmid

(2011) support Clark's view that technology serves at the pleasure of instructional design, pedagogical approaches, and teacher practices.

Even while people were debating the effectiveness of technology for education, technology leaders created programs to promote technology in the classroom. One such program started in 1985, Apple Classrooms of Tomorrow (ACOTTM) was a collaboration among public schools, universities, research agencies, and Apple Computer, Inc. (Ringstaff, Yocam, & Marsh,

1996). ACOT classroom students and teachers had access to a wide range of hardware and software (Ringstaff et al., 1996). The goal for technology in these classroom was tool for learning and a medium for thinking, collaborating, and communicating (Ringstaff et al., 1996), much like the goal of today. Early reports on ACOT found that many of the same first-order barriers to technology integration (Ertmer, 1999) existed during that period, with the major commentary devoted to resources and technical support (Sandholtz, Ringstaff, & Dwyer, 1990).

A later ACOT report states that teachers who had time to explore, to continue to learn new skills, and to plan lessons were more likely to change than those who did not (Sandholtz et al., 1990).

Similarly, we find that this holds true in the research today, as discussed later this chapter. More recently, Apple extended the program Apple Classrooms of Tomorrow—Today (ACOT2), wherein goal were to help high schools get closer to creating the kind of learning environment this generation of students’ needs, wants, and expectations (Apple, 2008). Rolled out in three phases, ACOT2 focused on designing classrooms for 21st Century learning, including innovation and creativity, social and emotional connections, and ubiquitous access to technology (Apple,

2008).

History of Technology Integration Focused Teacher Education

The history of educating prospective teachers on technology integration has a nearly seventy-year history. The OTA (U.S. Congress Office of Technology Assessment, 1995)

reported that beginning in the 1950s up to the 1970s programs to form teacher institutes, improve materials and training, and reform teacher education courses were sponsored by the national government. From those beginnings, Willis, Thompson, and Sadera (1999) explain that graduate courses for technology integration emerged in the late 1970s to early 1980s. Describing these programs as educational computing courses employed at the graduate level, Willis et al. (1999) explains that as technology grew the need developed for more specialized courses to develop how teacher implemented technology into teaching and learning.

Finding a need for specialized look at technology for teaching and learning, several organizations began focusing on the topic. In 1983, ISTE established a special interest group for teacher educators interested in the use of computers, and that group published the Journal of

Computing in Teacher Education, the first dedicated to scholarly research and professional practice in of technology integration (Willis et al., 1999). The next major step for researchers and educators came when the organization the Society for Information Technology and Teacher

Education (SITE) held the first national conference on the topic in 1990, and began publishing the Journal of Technology and Teacher Education (JTTE) (Willis et al., 1999). The number of outlets addressing the topic has grown from these beginnings.

Around the same period, policy makers began focusing on developing components of teacher education programs focused on technology integration. In 1988, Power On! New Tools for Teaching and Learning, conducted by the Office of Technology Assessment (OTA), reported on problems and issues surrounding efforts to increase the use of technology in schools, including teacher education (Bakir, 2016; Willis et al., 1999). Bakir (2016) explains the importance of the study came from the recognition that teacher education programs were not doing enough to promote technology skills in teachers. That report highlighted the programs

available from federal funds (e.g., teacher certifications, summer institutes, etc.), but acknowledged the trend to focus on inservice teachers, a trend that did not promote long term teacher quality (U.S. Congress Office of Technology Assessment, 1995). The OTA recommended funding programs that were higher intensity, longer term, and provided exposure to modern technologies (U.S. Congress Office of Technology Assessment, 1995). Following this recommendation, the first national technology, entitled Getting America’s Students Ready for the

21st Century: Meeting the Technology Literacy Challenge, stressed the importance of teacher education (Bakir, 2016). This plan promised training for teachers, access to hardware and software, and Internet connectivity (U.S. Department of Education, 1996). Future plans (2000,

2004, and 2010) continued the trend of national mandates (Bakir, 2016).

Akin to the ACOT program describe in the prior section, several teacher education initiatives for technology integration fund and resources supported this movement toward improving teacher education. The first such funding, a five-year, $2 billion initiative called

Technology Literacy Challenge Fund (TLCF) in 1997, provided for the goals of the 1996 technology plan (Bakir, 2016). In a study of the first year of funding, Kirshstein et al. (2000) found funds were used to support access to resources at the university level. Subsequent to the

TLCF, the Enhancing Education Through Technology program (EETT), a program aimed to improve student achievement in schools through technology integration, began in 2002 with the focus on using empirically-based methods to train and develop curricula for prospective teachers

(Bakir, 2016). About this same time, the Department of Education founded the Preparing

Tomorrow’s Teachers to Use Technology (PT3) grant program in 1999 for funding higher- education institutions, state agencies, school districts, and non-profit organizations efforts to preservice teachers’ technology-integration experiences (Bakir, 2016).

Modern focus on preparing future teachers take many directions. Kay (2006) identified ten strategies that are used currently. Those strategies included: a single course; mini-workshops; embedded integration into methods courses; using multimedia; collaborative methods between preservice teachers, mentor teachers, and faculty; field experience; focusing on education faculty; mentor teachers in PK12 settings; and improving access to software, hardware, and/or technical support (Kay, 2006). Each of these methods were shown by Kay to be employed throughout research on technology integration and teacher education.

The proceeding is a brief glimpse at the trends of policy makers and educators to increase the training of preservice teachers in technology integration. Even this brief look illustrates the focus of teacher preparation programs and national policies in supporting the use of technology by teachers. The historical view of teacher education practices, educational technology, and in turn technology integration, aids in situating the topic under consideration in this report. Most specifically, the debate around media-method may be the most influential consideration when examining PK12 technology integration. While Clark articulates a vital argument for the field of educational technology, it is important to note that the criticisms he levies are not completely vital to the research here. While the whole-hearted acceptance of technology may not be the best academic stance, the potential for technology integration is worth exploring. What does this integration look like? The following section defines technology integration, provides a working view of technology integration in PK12 classrooms, and discusses current standards initiatives, and practices for integration.

Defining and Illustrating PK12 Technology Integration

PK12 Technology Integration: A Definition

Defining technology integration may best be conceptualized by situating such practices within the domain of educational technology. Working outward from a formal definition, AECT

defines the field of educational technology as “the study and ethical practice of facilitating learning and improving performance by creating, using, and managing appropriate technological processes and resources” (p. 1). Breaking down this definition for its application in PK12 classrooms, the goal of this study is to better understand the practice of technology integration.

The practice of technology integration can vary greatly depending on the nature of the learning being enacted. Harris and Hofer (2009) identify eight continua that teachers should consider while preparing to integrate technology. These include: the focus of the lesson (i.e., teacher- to student-centric), convergent to divergent thinking, level of experience with the content, level of cognitive depth, plan duration, amount of lesson structure, grouping, and resources (Harris & Hofer, 2009). Additionally, the activity or practice goal may play a role in determining the type of technology integration. Building upon the 1994 AECT definition of educational technology, Reiser (2001) identifies design, development, utilization or implementation, management, evaluation, and analysis as avenues for the application of technology in education. The continua design features coupled with the performance tasks begin to address AECT’s definition in more precise terms that coincide with the research here.

Aside from the process of teaching and learning with technology, another consideration is the need for knowledge of the use of technology. Dale (1946) warned:

The use of these materials calls for more than a mastery of the mechanics and for more than an understanding of their power as teaching techniques. The teacher must also have a sense of proportion. She must be clear about her purposes and their relative values. For if the process (of audio-visual materials) is confused with the product (effective learning), audio visual materials may actually become the curriculum (p. 4).

This type of curriculum may best be categorized as technological content knowledge

(TCK)(Koehler & Mishra, 2009; Mishra & Koehler, 2006), Information and Communication

Technology (ICT) literacy (National Assessment Governing Board, 2014), or digital literacy

(Lankshear & Knobel, 2008). Each of these terms is defined in full later in this chapter. Yet, this warning, while having merit, may be less of a consideration in the modern era. The knowledge or ability of how to function in conjunction with technology has become a crucial outcome for education. As more and more the products of the world become intellectual properties, the ability to apply technology for better personal, academic, and professional products is critical for students to walk away from their education knowing how to do. Not unlike Dale’s warning,

Rotherham and Willingham (2010) advise that educators and policymakers must ensure that content is not shortchanged in the pursuit of skills, but the skills and knowledge are intertwined.

The intertwining of skills and knowledge is commonly referred to presently as 21st Century

Learning. The Partnership for 21st Century Learning (P21) illustrates the features of such learning for educators (Figure 2-2). As can be seen in the figure, 21st Century Learning is an amalgamation of real-world skills overarching educational content. This make the skills relevant both in and out of the classroom. As such, student development of these literacies, which for the purpose of this research will be named digital literacies, may very well be another vital outcome of a technology integrated classroom.

Figure 2-2. “Framework for 21st Century Learning” by the Partnership for 21st Century Learning, 2017 (http://www.p21.org/our-work/p21-framework). Copyright 2007 by the Partnership for 21st Century Learning (P21). Reprinted with permission.

What is meant by technology? National Assessment Governing Board (NAGB) defines technology as “any modification of the natural or designed world done to fulfill human needs or desires” (National Assessment Governing Board, 2014, pp. 1-5). This definition is quite broad, and could easily encompass a variety of items or processes. The focus within this research is digital technologies. Koehler and Mishra (2009) define digital technologies (e.g., computers, handheld devices, software applications etc.) as those tools are protean (i.e., usable in many different ways), rapidly changing, and opaque (i.e., the inner workings are hidden from users) (p.

61). Speaking to the debate between media and method, it becomes important to speak to the difference between media and technology. Reeves (1998) explains, “With respect to education, media are the symbol systems that teachers and students use to represent knowledge; technologies are the tools that allow them to share their knowledge representations with others”

(p. 5). The focus of this research are these digital technologies that are unique to the modern world, and subsequently education. The unique feature of these technologies is they are typically

not developed for the classroom, yet have a functionality that makes them powerful learning tools, as well as a real-world applicability.

Therefore, in the context of this research, technology integration is the practice of employing digital technologies by PK12 educators to facilitate the teaching and learning of both subject area content and 21st Century skills, including digital literacy. The next step is to understand the value of technology integration via a description of policy and practice.

PK12 Technology Integration in Practice

While having a clear understanding of what technology integration is, clarifying a stance of what technology integration in PK12 (PK12-TI) entails is vital for the conceptualization of this research. There are several levels of stakeholders impacted and impacting PK12-TI, including students, teachers, and policy maker. The following section provides an examination of the practice of PK12-TI for teachers and students, as well as a look at the standards and initiatives that shape those practices.

Policy makers at the highest level recognize the need for technology integrated learning.

The U.S. Department of Education (USDOE, or simply DOE) states unequivocally:

To remain globally competitive and develop engaged citizens, our schools should weave 21st century competencies and expertise throughout the learning experience. These include the development of critical thinking, complex problem solving, collaboration, and adding multimedia communication into the teaching of traditional academic subjects. In addition, learners should have the opportunity to develop a sense of agency in their learning and the belief that they are capable of succeeding in school (U.S. Department of Education (Office of Educational Technology), 2017, p. 10).

To put this learning into action, the USDOE enumerates five ways technology can improve and enhance learning:

1. Technology can enable personalized learning or experiences that are more engaging and relevant.

2. Technology can help organize learning around real-world challenges and project-based learning – using a wide variety of digital learning devices and resources to show competency with complex concepts and content.

3. Technology can help learning move beyond the classroom and take advantage of learning opportunities available in museums, libraries, and other out-of-school settings.

4. Technology can help learners pursue passions and personal interests.

5. Technology access when equitable can help close the digital divide and make transformative learning opportunities available to all learners (U.S. Department of Education (Office of Educational Technology), 2017, pp. 12-17)

Each of these points are supported in standards, frameworks, and research enacted by both government agencies and academic organizations. Exploring such work, one can conceptualize how PK12-TI learning might be operationalized.

To better understand the how PK12-TI may be operationalized, first understanding the influential standards and frameworks for technology integration, as well as the organizations that created them, provides useful context. First, the National Governors Association Center for Best

Practices (NGACBP) is a ”bipartisan organization of the nation's [USA’s] governors” (NGA,

2017) responsible for the creation of the Common Core State Standards (CCSS). The development of the CCSS, a set of clear college- and career-ready standards for PK12 in English language arts (LA) and mathematics, was led by the NGACBP, and created in conjunction with education commissioners, teachers, and the general public (National Governors Association

Center for Best Practices, 2010). Technology integration in these standards focuses on communication and publication across a variety of media for LA, and the use of data and tools for mathematics. The skills and knowledge focus within these standards are highly practical as the overall goals of the standards may suggest.

Beginning with the National Education Technology Standards for Students, or NETS*S, in 1998, the International Society for Technology in Education (ISTE) has evolved over three

iterations into the ISTE Standards for Students 2016. ISTE describes themselves as a worldwide network of educators dedicated to “empower[ing] learners to flourish in a connected world by cultivating a passionate professional learning community, linking educators and partners, leveraging knowledge and expertise, advocating for strategic policies, and continually improving learning and teaching” (International Society for Technology in Education, 2017a). The first iteration of standards created by ISTE, named the National Educational Technology Standards

(NETS), were published for students in 1998, and for teachers in 2000 (Barron, Kemker,

Harmes, & Kalaydjian, 2003). The student standards defined six technology competencies: basic operations and concepts; social ethical and human issues; technology productivity tools; technology communication tools; technology research tools; and technology problem solving and decision-making tools (Kelly, 2002). While focusing on practical applications of technologies, these first standards were highly focused on tool-based skills and knowledge focused on computing skills. The teacher standards at this time were designed to guide evaluation during teacher education and early careers of new teachers (Kelly, 2002). The teacher competencies included: technology operations and concept; planning and designing learning environments and experiences; teaching, learning, and curriculum; assessment and evaluation; productivity and professional practice; and social, ethical, legal, and human issues (Kelly, 2002). Approximately ten years after the first standards were published, the NETS for students and teachers received an update (in 2007 and 2008, respectively), and added standard sets for administrators (2009), coaches (2011), and computer science educators (2011) (International Society for Technology in

Education, 2017b). The new standards for students and teachers represented a shift away from a tools focus to a focus on 21st Century skills (e.g., communication and collaboration; research and information fluency; critical thinking, problem solving, and decision making, etc.)

(International Society for Technology in Education, 2017c). Through these updated standards, the goal for teachers was now to define the new skills and pedagogical insights educators need to teach, work, and learn in the digital age as they integrate technology and model technology use for students (International Society for Technology in Education, 2017d). Each of the standard sets also helps formulate ISTE’s “Essential Conditions” regarding effective technology integration (International Society for Technology in Education, 2017e). The next step in standards evolution came in 2013, when ISTE changed the name of the standards from NETS to the ISTE Standards to reflected the international reach of the standards, which were now being used by educators around the globe (International Society for Technology in Education, 2017f).

Recently, ISTE released the ISTE Standards for Students 2016 (ISTE16), the third iteration of standards from the organization, were developed by society members. The seven standard bands of this current iteration are described in Table 2-1. ISTE (2017g) argues that both ISTE16 and

CCSS stance towards anytime, anywhere access to a universe of facts means education as it’s always been done is not enough in the digital age, and new pedagogies must embrace all students’ innate drive to learn, create, collaborate, innovate, and think strategically. The evolution of technology integration in is reflected in ISTE’s standards. As leaders in PK12-TI,

ISTE started with teacher learning how to use technology, moved to using technology to learn, and are now focusing on transforming learning with technology (International Society for

Technology in Education, 2017h). Each stage has represented a significant shift in the focus of technology integration for PK12 education.

Table 2-1. Descriptors from ISTE Standards for Students 2016 from International Society for Technology in Education (2017h). Standard Band Overarching Description Empowered Learner Students leverage technology to take an active role in choosing, achieving and demonstrating competency in their learning goals, informed by the learning sciences. Digital Citizen Students recognize the rights, responsibilities and opportunities of living, learning and working in an interconnected digital world, and they act and model in ways that are safe, legal and ethical. Knowledge Students critically curate a variety of resources using digital tools to Constructor construct knowledge, produce creative artifacts and make meaningful learning experiences for themselves and others. Innovative Designer Students use a variety of technologies within a design process to identify and solve problems by creating new, useful or imaginative solutions. Computational Thinker Students develop and employ strategies for understanding and solving problems in ways that leverage the power of technological methods to develop and test solutions. Creative Students communicate clearly and express themselves creatively for Communicator a variety of purposes using the platforms, tools, styles, formats and digital media appropriate to their goals. Global Collaborator Students use digital tools to broaden their perspectives and enrich their learning by collaborating with others and working effectively in teams locally and globally.

Finally, a key framework supporting technology integration is the Technology Integration

Matrix (TIM). The TIM, first developed at the Florida Center for Instructional Technology

(FCIT) from 2003 to 2005 and updated in 2011, was created to help PK12 schools support students in learning 21st Century skills using pedagogically-centered language to describe effective technology integration (Harmes, Welsh, & Winkelman, 2016). While not nationally adopted, the TIM provides a solid framework for envisioning PK12-TI. The framework itself is comprised of two dimensions: levels of technology integration and characteristics of the learning environment. The levels of technology integration continuum ranges from entry, where the focus is on instruction, to transformation, where innovative use of technology is coupled with higher order learning activities in a local or global context (Harmes et al., 2016). The environment dimension is comprised of five interdependent characteristics (i.e., active, collaborative,

constructive, authentic, and goal-directed) designed to enable students to engage in higher-order thinking and focus on real-world skills (Harmes et al., 2016). The progression of the teaching and learning guided by the TIM can be seen in Figure 2-3.

Figure 2-3. Technology Integration Matrix Progression across levels of integration. Reprinted from Harmes et al. (2016).

Applying these standards and frameworks allows for the integration of technology by PK12 practitioners.

As stated above, the USDOE establishes that technology integrated learning is a critical aspect of modern education. Looking to the frameworks and standards highlighted above, the vision laid out by the DOE for enhanced learning can be clarified. First, the DOE Technology can enable personalized learning or experiences that are more engaging and relevant (U.S.

Department of Education (Office of Educational Technology), 2017). As noted in Figure 2-2,

PK12-TI learning at the higher end of the TIM allows for student ownership of the learning. This learning would be authentic, goal-directed, and deeply integrated with technology (Harmes et al.,

2016). ISTE says technology gives the power to work in real time with experts and peers across the globe, express our knowledge in a wide range of media, and disseminate our ideas to far-

flung, authentic audiences (International Society for Technology in Education, 2017g). Overall, this looks like students leveraging technology to take an active role in choosing, achieving, and demonstrating competency in their learning goals, or empowered learning (International Society for Technology in Education, 2017h).

The second goal of technology integrated learning outlined by the DOE means students face real-world challenges and project-based learning using a wide variety of digital learning devices and resources to show competency with complex concepts and content (U.S. Department of Education (Office of Educational Technology), 2017). The CCSS addresses this in a multitude of ways across both ELA and mathematics:

 They [students] are able to use technological tools to explore and deepen their understanding of [mathematic] concepts.

 [Students] apply the mathematics to practical situations, [and] use technology mindfully to work with the mathematics

 To be ready for college, workforce training, and life in a technological society, students need the ability to gather, comprehend, evaluate, synthesize, and report on information and ideas.

 They [students] are familiar with the strengths and limitations of various technological tools and mediums and can select and use those best suited to their communication goals.

 [Students] use technology, including the Internet, to produce and publish writing and to interact and collaborate with others (National Governors Association Center for Best Practices, 2010).

Each of these learning goals clearly relates to practical, real-world applications of technology. ISTE16 are deeply dedicated to addressing this type of learning, employing phrases such as “explore local and global issues,” “solving authentic problems,” and “work with open- ended problems” in describing learning outcomes (International Society for Technology in

Education, 2017h). Finally, the TIM devotes two row to the conditions of authentic and goal- directed (Harmes et al., 2016), which connects them to real-world, project-based learning.

Third, the DOE highlights technology’s ability to move learning beyond the classroom, and take advantage of opportunities in museums, libraries, and other out-of-school settings (U.S.

Department of Education (Office of Educational Technology), 2017). Another way to frame this is in the guide of collaboration. Again, the TIM devotes an entire row to the learning characteristic of collaboration. Collaborative learning may involve students working with other students in using technology to complete tasks, or using technology to collaborate with peers and experts outside of their classroom (Harmes et al., 2016). ISTE aims to make students creative communicators, global collaborators, and digital citizens (International Society for Technology in Education, 2017h). The benefits of this for students lies in the power to do things never done before, like work in real time with experts and peers across the globe, express knowledge in a wide range of media, and disseminate our ideas to far-flung, authentic audiences (International

Society for Technology in Education, 2017g). Additionally, the CCSS encourage student knowledge of the strengths and limitations of various technological tools and mediums, and the ability to select and use those best suited to their communication goals (National Governors

Association Center for Best Practices, 2010). Communicating and collaborating are essential processes for 21st Century leaning.

The final two directives from the DOE applies technologies to helping learners pursue passions and personal interests, and close the digital divide by making transformative learning opportunities available to all learners. The digital divide explains the educational gap between those who have technology as a resource, and those who do not. Hohlfeld, Ritzhaupt, Barron, and Kemker (2008) model three levels of digital divide along stages of infrastructure, classroom, and student (see Figure 2-4).

Figure 2-4. Levels of the digital divide. Adapted from Hohlfeld et al. (2008).

Presuming a solid infrastructure, the second and third stages of the divide are addressable through classroom practices. This leads to the empowerment of students for learning. ISTE describes the empowered learner as those who leverage technology to take an active role in choosing, achieving and demonstrating competency in their learning goals, informed by the learning sciences (International Society for Technology in Education, 2017h). CCSS encourage the use of technologies (e.g., screen-readers, speech-to-text, etc.) to allow for the widest possible range of students to participate fully from the outset and as permitting appropriate accommodations to ensure maximum participation of students with special education needs

(National Governors Association Center for Best Practices, 2010). The effective integration of technology in the PK12 classroom has the ability to widen or shrink the digital divide based on the practice of the teacher.

While some may dispute the value of technology integration, there is, at the very least, a strong policy push from the government and top educational organization to bring technology integrated learning to the PK12 classroom. Consequently, technology integration is not only a teacher’s prerogative, so much as an essential component of modern education. However, studies have shown that there are various barriers to technology integration both external and internal to

the teacher. In the next section, the various degrees of technology integration barriers are presented, along with an examination of the individual components comprising each and how they might be addressed through teacher education courses.

Barriers to Technology Integration in PK12 Classrooms

Technology integration in classrooms from prekindergarten to 12 can be a powerful agent in education. Yet, there are many classrooms across schools today wherein the teacher balks at the idea of technology integration. The goal of this research is understanding how teacher education courses for technology integration in the classroom can impact barriers to technology integration. Barriers are those things that prevent access or keeps things apart to prevent progress. In the context of this research, a barrier equates to the idea that any deficiency or absence of educational conditions promoting technology integration creates an environment wherein integration is hindered or negated. Ginsburg (2009) defines four stages of teacher development: apprenticeship of observation, preservice, induction, and inservice. Being the first stage of development as a professional educator, preservice teacher education courses devoted to technology integration may provide a foundational opportunity for teacher educators to impact the direction of technology integration in classrooms. The question is: What barriers to technology integration exist, and are addressable through teacher education? In the following section, barriers to PK12 technology integration are categorized, and set within a framework to contextualize those barriers to be addressed by technology integration courses.

The barriers to technology integration in a PK12 context is an often-researched topic in academic literature. Conducting a review of 123 empirical studies, Hew and Brush (2007) classified six major categories that, when missing or weak, may act as barriers: (a) resources, (b) knowledge and skills, (c) institution, (d) attitudes and beliefs, (e) assessment, and (f) subject culture (p. 226). Lack of resources may include any number of the following: technology,

appropriate access to available technology, time, and technical support (Hew & Brush, 2007).

Hew and Brush (2007) identified lack of technology knowledge and skills, technology- supported-pedagogical knowledge and skills, and technology-related-classroom management knowledge and skills as the second most discussed barrier. Third, institutional barriers included: leadership, school time-tabling structure, and school planning (Hew & Brush, 2007). Fourth, attitudes (i.e., specific feelings that indicate whether a person likes or dislikes something) and beliefs (i.e., premises or suppositions about something that are felt to be true) were the fourth most significant barrier in the literature (Hew & Brush, 2007). Assessments, specifically high- stakes testing, steers teachers towards lectures, and away from technology integration because of the additional time requirement for planning integrated lessons (Hew & Brush, 2007).

Additionally, the used of school technology resources for testing and/or test practice limits the access that teacher and students have to those resources for learning purposes (Hew & Brush,

2007). Finally, subject culture (i.e., the set of institutionalized practices and expectations which have grown up around a particular school subject, and shapes the definition of study of that subject) can cause teachers to avoid technology integration that seems incompatible with the norms of a subject culture (Hew & Brush, 2007). From their review, a tentative model was illustrated to demonstrate relationships between these categories (see Figure 2-5). As illustrated, the interactions of these categories can influence both direct effects, as well as creating indirect effects on technology integration. It is important to note that a couple of barriers (i.e., assessment and subject culture) only have indirect effects, which probably accounts for the relatively small percentage (5% and 2% frequency, respectively) of discussion in the identified research (Hew &

Brush, 2007). Support for each of these categories can be seen throughout technology integration literature.

Figure 2-5. Theorized relationships between technology integration barriers. Adapted from Hew and Brush (2007).

As part of their effort to define the high quality technology integration practice, ISTE establishes fourteen critical elements necessary to effectively leverage technology for learning, in other words “Essential Conditions” (International Society for Technology in Education, 2017e).

See Table 2-2 below for a description of the conditions. Once again, it is the deficiency or absence of these conditions that act as barriers to technology integration.

Table 2-2. Descriptions of ISTE’s (2017e) Essential Conditions. Condition Definition Shared Vision Proactive leadership develops a shared vision for educational technology among all education stakeholders, including teachers and support staff, school and district administrators, teacher educators, students, parents and the community. Empowered Leaders Stakeholders at every level are empowered to be leaders in effecting change. Implementation Planning All stakeholders follow a systematic plan aligned with a shared vision for school effectiveness and student learning through the infusion of information and communication technology (ICT) and digital learning resources. Consistent and Adequate Funding Ongoing funding supports technology infrastructure, personnel, digital resources and staff development.

Table 2-2. Continued. Condition Definition Equitable Access All students, teachers, staff and school leaders have robust and reliable connectivity and access to current and emerging technologies and digital resources. Skilled Personnel Educators, support staff and other leaders are skilled in the selection and effective use of appropriate ICT resources. Ongoing Professional Learning Educators have ongoing access to technology-related professional learning plans and opportunities as well as dedicated time to practice and share ideas. Technical Support Educators and students have access to reliable assistance for maintaining, renewing and using ICT and digital learning resources. Curriculum Framework Content standards and related digital curriculum resources align with and support digital age learning and work. Student-Centered Learning Planning, teaching and assessment all center on the needs and abilities of the students. Assessment and Evaluation Teaching, learning, leadership and the use of ICT and digital resources are continually assessed and evaluated. Engaged Communities Leaders and educators develop and maintain partnerships and collaboration within the community to support and fund the use of ICT and digital learning resources. Support Policies Policies, financial plans, accountability measures and incentive structures support the use of ICT and other digital resources for both learning and district/school operations. Supportive External Context Policies and initiatives at the national, regional and local levels support schools and teacher preparation programs in the effective implementation of technology for achieving curriculum and learning technology (ICT) standards.

A further examination of the technology integration literature supports provides further agreement with the barriers suggested by Hew and Brush and ISTE, as well as contributes additional factors for consideration. Framing each barrier within the Hew and Brush model provides a concise look at the barriers discussed in the literature. The following table summarizes the research on technology integration barriers. Access (Bingimlas, 2009;

Blackwell, Lauricella, Wartella, Robb, & Schomburg, 2013; Lowther, Inan, Strahl, & Ross,

2008; Reid, 2014) and technology, curricular, and other resources (Bingimlas, 2009; Inan &

Lowther, 2010; Keengwe, Onchwari, & Wachira, 2008; Lowther et al., 2008; Reid, 2014) are grouped here under the resources heading. As discussed earlier in this chapter, Hohlfeld et al.

(2008) define three digital divide levels, wherein Level 1 might correspond with resources here,

while Level 2 equates with access. Other facets of integration barriers could certainly fall under the resource heading, as well. Technical support, engaged communities, and support policies as part of ISTE’s Essential Conditions bridge the gap between what Hew and Brush deem resources and institution, because of the call for resources and institutional policies accompanying technology support. Lack of technical support undoubtedly has a place in the research as a potential barrier (Bingimlas, 2009; Inan & Lowther, 2010; Keengwe et al., 2008; Lowther et al.,

2008; Ritzhaupt et al.). As in the Hew and Brush article, other researchers identified leadership as a key aspect of institutional barriers (International Society for Technology in Education,

2017e; Keengwe et al., 2008; Lowther et al., 2008; Reid, 2014). Along with school planning

(implementation planning; International Society for Technology in Education, 2017g), environment and faculty (Blackwell et al., 2013; Inan & Lowther, 2010; Lowther et al., 2008;

Reid, 2014; Ritzhaupt et al.), as well as a shared vision and ongoing professional development

(International Society for Technology in Education, 2017e) or training (Bingimlas, 2009; Inan &

Lowther, 2010; Keengwe et al., 2008; Reid, 2014; Ritzhaupt et al., 2012), could each comprise an aspect towards institutional barriers. Attitudes and beliefs, (Bingimlas, 2009; Blackwell et al.,

2013; Inan & Lowther, 2010; Lowther et al., 2008), and either content or technological knowledge bases (Bingimlas, 2009; Inan & Lowther, 2010; Lowther et al., 2008) are each examined in literature, and compose their own classifications. Also worth nothing, as a precursor to ISTE’s Essential Conditions, Dias (1999) lists change (i.e., the adoption of new teaching tools and a new stance on how to teach) in addition to time, training, resources, and support as a barrier. Change is probably best categorized as a component of beliefs and attitudes. One category not defined by Hew and Brush is that of teacher demographics. Age, gender, and/or

years of teaching have been identified in the research (Inan & Lowther, 2010; Ritzhaupt et al.,

2012) as other factors influencing technology integration.

Table 2-3. A summary list of the barriers affecting technology integration in PK12 classrooms. Barrier Article

Duran (2001) Duran

(2017e)

(2011) Reigeluth An and (2009) Bingimlas (2013) al. et Blackwell and Kariuki, Turner, Franklin, (2009) Yildirim and Yildirim, Goktas, (2010) Lowther and Inan ISTE (2008) al. et Keengwe (2016) al. Liu et (2008) al. et Lowther (2014) Reid (2012) al. et Ritzhaupt Resources ✓ ✓ - ✓ ✓ ✓ ✓ ✓ ✓ ✓ ✓ ✓ Knowledge and Skills ✓ ✓ - - ✓ ✓ ✓ - ✓ ✓ - ✓ Institution ✓ ✓ ✓ - ✓ ✓ ✓ ✓ ✓ ✓ ✓ ✓ Attitudes and Beliefs ✓ ✓ ✓ ✓ - ✓ - - ✓ ✓ - - Assessment ✓ - - ✓ ------Subject Culture ✓ ------Teacher Demographics - - - - - ✓ - - ✓ - - ✓

To discuss barriers for PK12-TI, Hew and Brush (2007) conceptualized their discussion on the topic around Ertmer’s (1999) classifications for first- and second-order barriers to change.

The research of Ertmer was extended to this meta-analysis, but with an adaptation to the definitions for each order of barriers. Ertmer (1999) defined first-order barriers as those obstacles that are extrinsic to teachers. These barriers are described in terms of missing or inadequate resources in teachers' implementation environments, making these barriers easy to measure and relatively easy to eliminate (Ertmer, 1999). Two assumptions result from this view of first-order barriers. First, once adequate resources were obtained that integration would follow, or the implementation process could not even begin until all the necessary resources were in place

(Ertmer, 1999). In the research of Ertmer (1999) first-order barriers included resources, access, training, and support; while in addition Hew and Brush (2007) included institution, subject culture, and assessment to the first-order.

The second-order barriers to technology integration, defined by Ertmer (1999), were those that interfere with or impede fundamental change. These barriers are rooted in teachers' underlying beliefs about teaching and learning, and may not be apparent to others or even to the teachers themselves (Ertmer, 1999). Second-order barriers may cause more difficulties than first- order barriers, because they are less tangible, more personal, more deeply ingrained than first- order barriers. Hew and Brush (2007) cite attitudes and beliefs, as well as knowledge and skills as falling under the second-order heading. To help align the views of Ertmer with Hew and

Brush, the Hew and Brush categories are grouped below the appropriate order heading in the image below (Figure 2-6).

Figure 2-6. Aligning Hew and Brush (2007) barriers with Ertmer (1999) barrier degrees.

Building on the work of Ertmer, as well as Hew and Brush, the research in this dissertation provides a deep look into the distinctions between attitudes and beliefs, conceptual and practical knowledge of technology and its integration in PK12 education, but most important the impact that teacher education course can play in removing them as barriers for integration by future teachers. Extending on the view of first- and second-order barriers to change, this research shifts the definition of each order slightly. First-order barriers are no longer simply constrained to those barriers extrinsic to teachers. First-order barriers are defined here as those obstacles that are unalterable as a result of teacher education courses. These barriers could be either intrinsic or extrinsic to the educator. Similar to Ertmer’s views on the barriers to change, and by extension

Hew and Brush’s model; this research considers resources, institution, subject culture, and assessment as aspects related to first-order barriers. In contrast to prior research, this research additionally includes those elements comprising teacher demographics (e.g., age, gender, years of experience, etc.). While these elements are clearly intrinsic aspects of the teacher’s identity, they are in no way subject to change through education. As with the Hew and Brush model, teacher demographics would likely shape to some extent the attitudes, beliefs, and knowledge of the teacher, but cannot be altered using methods discussed here.

The understanding the impact of TECTI on second-order barriers to technology integration was the primary focus of this research. Akin to Ertmer (1999) research, these barriers are those that interfere with or impede fundamental change, and which are rooted in teachers' underlying beliefs about teaching and learning, and may not be apparent to others or even to the teachers themselves since they are less tangible, more personal, more deeply ingrained than first- order barriers. Second-order barriers to technology integration are attitudes and beliefs, as well as knowledge and skills. Hew and Brush (2007) define attitudes as specific feelings that indicate

whether a person likes or dislikes something, in this case teachers liking or disliking the use of technology. Meanwhile, beliefs were defined as premises or suppositions about something that are felt to be true, which included educational beliefs about teaching and learning (i.e., pedagogical beliefs), and their beliefs about technology (Hew & Brush, 2007). In addition to attitudes and beliefs, Hew and Brush (2007) also viewed knowledge and skills as falling under the second-order barrier heading. In that study, the lack of specific technology knowledge and skills, technology-supported-pedagogical knowledge and skills, and technology-related- classroom management knowledge and skills were identified as a major barrier to technology integration (Hew & Brush, 2007). While working well as a model, Hew and Brush’s research was re-conceptualized to a degree to shape the research here.

As visualized above, Hew and Brush’s research into technology integration barriers explored first-order barriers (i.e., resources, institution, assessment, and subject culture), and second-order barriers (i.e., attitudes/beliefs and knowledge/skills) to technology integration. While agreeing with the alignment of the barriers, the conceptualization of the barrier categories differs in this research. Teacher attitudes and teacher beliefs comprise different, yet interrelated, categories.

Furthermore, knowledge and skills are viewed in the context of conceptual knowledge and practical knowledge, again separating them into different classifications from Hew and Brush’s model. A new model was formulated to guide the present study (Figure 2-7). As modeled, rather than attitudes and beliefs along with skills and knowledge, the outcomes under consideration are now attitudes, beliefs, and knowledge with knowledge being comprised of two aspects: conceptual knowledge (i.e., technological pedagogical, technological content, and TPACK) and practical knowledge (i.e., ICT literacy). Each of these second-order barriers are impacted as before by influencing factors of within both orders. However, because the first-order barriers are

contextual to the school setting, they are viewed as being influences only for inservice teachers for the purposes of this study. Therefore, the primary focus was on the three second-order categories. In the following section, the three outcome variables are conceptualized, and potential measures for the variables are identified.

Figure 2-7. Visualization of framework guiding current meta-analysis.

Examination of the Outcome Variables

As established, technology integration faces barriers in the classroom both internal and external of the teacher. However, those internal to the teacher may be changeable though education prior to the teacher entering the classroom. Attitudes, beliefs, and knowledge are the outcomes under consideration within this meta-analysis. As such, this chapter portion is devoted first to clearly defining the nature of each construct, then exploring how attitudes, beliefs, and knowledge related to technology and technology integration have been measured.

Understanding Attitudes, Beliefs, and Knowledge

Attitudes

Attitude is the first of the outcome variables to be described. Attitudes toward technology have consistently shown to be predictors of technology integration behavior (Buckenmeyer,

2010; Ertmer, 1999; Goktas et al., 2009; Hew & Brush, 2007). The APA Dictionary of

Psychology defines attitude as:

[A] relatively enduring and general evaluation of an object, person, group, issue, or concept on a dimension ranging from negative to positive. Attitudes provide summary evaluations of target objects and are often as attention to be derived from specific beliefs, emotions, and past behaviors associated with those objects (Attitude, 2015).

Using this as a foundation, a conception of attitudes for this study begins. The definition of attitude here begins with the target of the attitude, which is technology. To some extent, the attitude object (i.e., any people, social groups, policy positions, abstract concepts, or physical objects that has an attitude associated with it; Attitude object, 2015) could also be thought of as technology integration. However, the focus here will strictly be on technology itself. Attitudes are made up of cognitive, affective, and performance components; as well as various motivational functions (i.e., knowledge, ego-defense, value-expression, etc.) (Aiken, 2002).

Attitude research highlights the connection between affect and attitude (e.g., Clore and Schnall

(2005); Forgas (2008)). While defining each of the components, Triandis (1971) explained the affective component equates with “the emotion which charges the [attitude]” (p. 3). In the context of this research, the affective nature of the attitude aligns it as a mental attitude (i.e., a subjective or mental state of preparation for action), rather than a motor attitude, or an outward posture (Allport, 1967). Therefore, a technology attitude is an affective mental state related to technology, and its use. These attitudes can be both positively and negatively charged. To better understand the positioning of positive and negative attitudes, Triandis (1971) plotted them along

two dimensions: a) positive vs. negative affect; and b) seeking vs. avoiding contact (Triandis,

1971, p. 12). The dimensional differences result in observable behaviors in a person. Examples of behaviors across the dimensions are presented in Figure 2-8.

Figure 2-8. Attitudinal dimensions. Adapted from Triandis (1971, p. 13).

Bodur, Brinberg, and Coupey (2000) suggest affect and attitude are separate constructs and that affect has a direct and significant effect on attitude. This argument can be supported by the

Triandis model. However, Bodur et al. (2000) acknowledge that the long supported position of research is that affect and attitude are a united construct. Therefore, the direct connection between affect and attitude was the position here. Considering this connection between affect and attitude, how do technology related attitudes manifest?

To further clarify the attitudes within this research, it is necessary to explore the multitude of technology attitude related constructs defined in literature. Various technology attitudes may include computer (technology) anxiety (Heinssen et al., 1987; Meuter et al., 2003), technophobia (Campion, 1989; Rosen & Weil, 1995b), technophilia (Campion, 1989), computer

(technology) confidence/self-efficacy (Lee & Lee, 2014; L. Wang et al., 2004), technology acceptance (Davis, 1993; Davis et al., 1989; Venkatesh et al., 2003), technology adoption behaviors (Lin, 2003; Parente & Prescott, 1994), perceived usefulness of technology (Davis,

1993), and technostress (Ennis, 2005; Ragu-Nathan et al., 2008; K. Wang et al., 2008).

Following is a brief description of each attitude, its causes, and its context to this research.

Heinssen et al. (1987) defined computer anxiety as is a negative affective response to computers. Meuter et al. (2003) defined technology anxiety defined as fear or apprehension people feel when considering use or actually using technology that leads to excessive timidity in using computers, negative comments against technology, attempts to reduce the amount of time spent using technology, and even the avoidance of technology. Because of the differing nature of technologies more currently, extending the definition to include all technologies extends the value of the research. Technology anxiety is a function of fear and apprehension, intimidation, hostility, and/or worries that one will be embarrassed, look stupid, or even damage technology

(Heinssen et al., 1987). It is the affective response that precisely categorizes technology anxiety as an attitude for this study. Technology anxiety can be both personal and educational use, because an anxiety in personal technology use would feed into an educational use.

Technophobia and technophilia are opposing constructs. Rosen and Weil (1995b) defined technophobia as anxiety about current or future interactions with technology; negative global attitudes about technology or its societal impact; and/or (c) specific negative cognitions or self- critical internal dialogues during actual or future technology interaction. As such, technophobes would criticize both processes and outcomes of using technology (Campion, 1989). Conversely, technophilia exists in those who favor technology, and commend technology processes and the outcomes (Campion, 1989). The relation of these two views to this research lies in the approach.

Technophobia would be a construct to change through education, while technophilia would only need to be promoted.

Recent research into computer (technology) confidence/self-efficacy related to technology integration has focused specifically on technology integration by teachers (Lee &

Lee, 2014; L. Wang et al., 2004). Bandura (2006) explained that perceived self-efficacy is a judgment of capability to execute given types of performances; outcome expectations are judgments about the positive and negative physical, social, and self-evaluative outcomes that are likely to flow from such performances. Successful performance attainment, observing the performances of others, verbal persuasion indicating certain capabilities, and physiological states that judge capability, strength, and vulnerability (Bandura, 1986). This efficacy affects behavior not only directly through its influence as to whether people think erratically or strategically, optimistically or pessimistically, but by its impact on other determinants such as goals and aspirations, outcome expectations, affective proclivities, and perception of impediments and opportunities in the social environment (Bandura, 2006). Therefore, while Bandura describes self-efficacy as a “belief,” because of the described interaction between affect and behavior, self- efficacy can be grouped here as an attitude.

Technology acceptance uses theory of reasoned action to specify a causal linkage between two beliefs: perceived usefulness and perceived ease of use, and users’ attitude intentions and actual technology adoption (Davis, 1993; Davis et al., 1989). The related construct perceived usefulness, which Davis (1989) defined as a user’s subjective probability that using a technology will increase job performance, was also considered an attitude here. In the article,

Davis called this construct a belief. However, the grouping here with attitudes stems from the subjective nature of the belief. Within the technology acceptance model, a relationship is

proposed between a person’s attitude towards using technology and their behavioral intention to use. Davis et al. (1989) suggests relationship represented in the technology acceptance model implies that all else being equal people form intentions to perform behaviors toward which they have positive affect (p. 986). Later research showed three direct determinants of intention to use

(i.e., performance expectancy, effort expectancy, and social influence) and two direct determinants of usage behavior (i.e., intention and facilitating condition) (Venkatesh et al.,

2003). Technology acceptance attitudes have been shown to have validity in an educational context (Teo, 2009). Thus, technology acceptance, and the related perceived usefulness, must both be considered as attitudes here.

Technology adoption factors, specifically related to the user, may be yet another indicator of technology attitudes. Lin (2003) lists four factors influencing technology adoption:

 predisposed personality traits that make the audience receptive to the idea of innovation adoption (e.g., risk tolerance)

 self-actualization need for adoption (e.g., for work or pleasure)

 beliefs about one’s ability to adopt and use a technology innovation with computers

 beliefs and attitudes about the rationale for innovation adoption (p. 350)

While again blurring the line between beliefs and attitudes, multiple point wither mention attitudes or use terminology connected to attitudes discussed earlier in this chapter (e.g., ability beliefs relate to confidence/self-efficacy). Additionally, some personality traits identified by Lin

(e.g., willingness to take risks or entertain new ideas related to technology) may be indicative of a positive affective stance. Therefore, technology adoption is seen as related to technology attitudes.

Attributed to research by Craig Brood in the 1980s, technostress is a modern disease of adaptation caused by an inability to cope with new computer technologies in a healthy manner

experienced by individuals due to the use of technology (Ennis, 2005; Ragu-Nathan et al., 2008;

K. Wang et al., 2008). Technostress is seen as being caused by one’s attempts to deal with evolving technology, and the changing physical, social, and cognitive responses demanded by its use (Ragu-Nathan et al., 2008). According to K. Wang et al. (2008), technostress relates to technophobia, cyberphobia, computerphobia, computer anxiety, computer stress, negative computer attitudes, and other similar terms in literature. The general emotional nature of stress coupled with the supported relation to other attitude constructs defined in this chapter leads to the inclusion of technostress and another attitude factor.

To support the visualization of technology attitudinal contexts, the Triandis chart was adapted to indicate the affective positioning of each attitude (Figure 2-9).

Figure 2-9. Technology attitudes.

On the new graph, each affective attitude is graphed to demonstrate a) where it falls along the affect dimension, and b) approximate likely behaviors resulting from the attitude. Some aspects

have clearly defined locations. For example, each technostress, technology anxiety, and technophobia are negative on the affect scale, and thereby could/would lead people to avoid technology. Technostress is closest to the origin conceptually, because the inability to cope with technology would be the impetus of anxiety. Certainly, stress can result from anxiety, yet this is imagined as a different construct because the stress is not directly from technology. People with technology anxiety fall between those with technostress and technophobes. It can be assumed that technology anxiety would come before technophobia, and that technophobes would have a high degree of anxiety when forced to use technology. A similar relationship is assumed between technology adoption and technophilia. You may adopt technology, but not have a special affinity for it. Yet, if you are a technophiliac, the presumption is that you readily adopt technology without hesitation. Perceived usefulness of technology runs along the behavior dimension.

Strong perception of usefulness would probably lead to seeking technology, and conversely a low perception would lead to avoidance. This may not be completely accurate (e.g., some people may agree technology is very useful, but not actively seek it out), so there may not be a direct correlation as illustrated. Finally, technology confidence/self-efficacy can be viewed as bridging both the lower left and the upper right quadrants. Again, this works on the assumption that high levels of those constructs would lead people towards technology, while low levels would cause people to avoid it.

Technology attitudes can be measured along a variety of dimension or criteria. In a meta- analysis of computer anxiety, Chua, Chen, and Wong (1999) examined the relationship of this anxiety among studies focusing on correlates (e.g., gender, age, etc.). More commonly in attitude studies, researchers Likert scales. For example, technology anxiety (Rosen & Weil, 1995a), self- efficacy (Hasan, 2003), and technology adoption (Birch & Irvine, 2009) have each been

measured using Likert scales. These, or other, methods of empirically measuring attitudes fit within this meta-analysis, as long as the tool can be distinguished from beliefs, as discussed next.

How do attitudes and beliefs differ for this research? Pajares (1992) argued that teacher beliefs specifically a blend of teacher efficacy, epistemological beliefs, about causes of teachers’ or students’ performance, about perceptions of self and feelings of self-worth, self-efficacy, and educational beliefs about specific subjects or disciplines. While some of these pieces have been labeled here as attitudes, and others not, the intense inclusion of attitudes in teacher belief suggests that separating one from the other may be impossible. However, unlike attitudes make up of cognitive, affective, and performance components, Aiken (2002) explains that beliefs are similar to opinions, because of the inherent judgement and acceptance of something as fact (p.

6). Fishbein and Ajzen (1975) distinguish between beliefs and attitudes in that the former is a subjective likelihood of performance, while the latter is an affective evaluation (p. 233). Eagly and Chaiken (1993) suggest that beliefs are more closely related to the cognitive component of attitudes suggested by social scientists in prior research. This suggests that one separating condition is the nature of the connection to attitude objects with attitudes being more emotional, while beliefs are grounded in thought. However, which influences the other? Some researchers have found that beliefs are determinates for attitudes (Avramidis & Norwich, 2002; Bodur et al.,

2000; Davis et al., 1989). Meanwhile, the APA Dictionary of Psychology defines a belief as an evaluative association with an attitude object (i.e., any target of judgment that has an attitude associated with it) (Attitude object, 2015; Belief, 2015). Regardless of the flow of the relationship, if one causes the other, then that indicates that while influencing each construct is distinct from the other. Clearly defining the flow of the attitude-belief relationship is outside the context of this research. As such, attitudes and beliefs will be considered separate from the other

within this context. Furthermore, the nature of beliefs for this research is connected directly to technology integration, as well as teaching and learning. Because of this, the term belief has a unique connotation beyond the traditional concept. A complete definition of beliefs related to this research can be found in the following section.

Beliefs

Beliefs, while closely intertwined with attitudes, are intellectualized as separate constructs for this research. Studies have shown that beliefs about the practice of teaching are a significant determinant in explaining why teachers adopt computers in the classroom (Hermans,

Tondeur, van Braak, & Valcke, 2008). Yet, how exactly do beliefs related to technology manifest? Peggy Ertmer named these beliefs teacher pedagogical beliefs (Ertmer, 2005). She calls such beliefs the “final frontier” for technology integration (Ertmer, 2005), and the interplay between these beliefs and technology integration as both a “critical relationship” to understand

(Ertmer, Ottenbreit-Leftwich, Sadik, Sendurur, & Sendurur, 2012). Using Ertmer’s (2005) conception of teachers’ pedagogical beliefs as a guide, the following explores the nature of teacher beliefs as they relate to technology integration.

Teacher pedagogical beliefs are those educational beliefs in teachers about teaching and learning, and the corresponding beliefs about how technology enables them to translate those beliefs into classroom practice (Ertmer, 2005). In her research, Ertmer frames her definition around Pajares’ (1992) attempt to clarify the “messy construct” of teacher beliefs. Pajares (1992) identified four features characteristic of beliefs: existential presumption, alternativity, affective and evaluative loading, and episodic structure (p. 309). How does each impact teacher beliefs?

Existential presumptions are the incontrovertible, personal truths everyone holds, and they are deeply personal, rather than universal, and unaffected by persuasion. (Pajares, 1992, p.

309). An alternate way to state this is to say that they are the truths people have about the nature

of teaching and learning. As part of her definition of beliefs related to learning to teach,

Richardson (1996) explained that beliefs describe propositions accepted as true by the individual holding those beliefs, and that beliefs are a psychological concept differing from knowledge which implies epistemic warrant. All philosophies of education are a dance between epistemology and ontology. Schuh and Barab (2008) explain epistemology defines how humans come to know what exists (i.e., the origins, nature, methods, and limits of human knowledge), while ontology defines what is real in the world (i.e., the nature of being and reality). In 2009, the OECD published indices related to teachers’ beliefs about teaching. The first addressed direct transmission beliefs about teaching, that included:

 Effective/good teachers demonstrate the correct way to solve a problem.

 Instruction should be built around problems with clear, correct answers, and around ideas that most students can grasp quickly.

 How much students learn depends on how much background knowledge they have; that is why teaching facts is so necessary.

 A quiet classroom is generally needed for effective learning (OECD, 2009).

Alternately, the second focused on constructivist beliefs about teaching:

 My role as a teacher is to facilitate students’ own inquiry.

 Students learn best by finding solutions to problems on their own.

 Students should be allowed to think of solutions to practical problems themselves before the teacher shows them how they are solved.

 Thinking and reasoning processes are more important than specific curriculum content (OECD, 2009).

Each of these views are grounded in the teacher’s epistemological views. To connect these existential presumptions to the present research, requires the inclusion of technology. The view must be that in some way technology enhances these practices. However, the beliefs of the teachers related to technology integration can both help and hinder. Research has shown that

while teacher beliefs about the practice of teaching are a significant determinant in integration, that constructivist teacher beliefs could predict classroom use, while traditional teacher beliefs seem to have a negative impact on integration (Hermans et al., 2008). This may suggest that the call becomes for the promotion of constructivist views to support the use of technology.

Labeled alternativity, Pajares (1992) explained that sometimes teachers attempt creating an ideal situation, which may differ from reality, for teaching and learning. To integrate technology, this ideal condition must again involve the use of technology to create transformative learning experiences. However, it may be valuable not to accept technology integration as the ultimate condition for learning. Selwyn (2015) argues that academic research and writing often does not address adequately the social, political, economic and cultural complexities of technology and education. The recommended lens of research being one of objectivity; producing detailed and contextually-rich analyses; engaging in objective evaluation; and requiring investigation in terms of its positives, negatives and all areas in-between (Selwyn,

2015). While this is a valid concern, the goal must still be technology integrated learning, but with this critical eye.

Pajares’ stance on the affective and evaluative loading related to teacher beliefs is that all human perception is influenced by generic knowledge structures (e.g., schemata, constructs, information, beliefs, etc.), but the structure itself is an unreliable guide to the nature of reality because beliefs influence how individuals characterize phenomena, make sense of the world, and estimate covariation (Pajares, 1992, p. 310). He goes on to say that cognitive knowledge must also have its own affective and evaluative component (Pajares, 1992). Within the present study, while it is agreed that there is potentially an affective piece to beliefs because of the intimate

relationship with attitudes, that the evaluative, cognitive aspect is the primary role for beliefs related to technology integration.

Finally, Pajares (1992) suggested that teacher beliefs draw power from previous episodes or events that colored the comprehension of subsequent events, (i.e., episodic structure).

Meaning, positive or negative experiences related to technology integration can help or hinder the belief that technology plays a positive role in education. Some research (e.g., Ertmer, 2005;

L. Wang et al., 2004) highlights the value in observation for shaping views on technology integration, thereby suggesting that the episodic structure suggested by Pajares can impact teacher beliefs in this area. More discussion of this structure is discussed later in the chapter.

Building from Pajares, Ertmer (2005) defines teachers’ beliefs as those related to teaching and learning, and names them pedagogical beliefs. Similarly, when referring to beliefs within this study, the definition equates with those pedagogical beliefs. Yet, the pedagogical beliefs presently are only those related to the view of the teacher on teaching and learning with technology. Angers and Machtmes (2005) model such beliefs as the evaluative judgement that technology is a tool that adds value to lessons, as well as technology adds value to student learning and motivation. Zhao and Cziko (2001) content to support technology integration that teachers must believe that technology can more effectively meet higher-level goals, will not cause disturbances to other goals, and they have or will have sufficient ability and resources to use technology. The pedagogical beliefs a teacher holds regarding technology integration if verbalized may sound akin to “Technology has a value to my students as both learning and practical tool both now and in the future,” the polar opposite of this statement, or something along a continuum between these two extremes. The full combination of evaluation for pedagogical choices related to technology integration is summarized here as beliefs.

Why is it important to understand beliefs, as well as attitudes? Maio and Haddock (2014) explain:

Beliefs may be even more important for attitudes toward a variety of important issues. For instance, people’s attitudes toward car use can be influenced by their beliefs about climate change; people’s attitudes toward different politicians may be based on their beliefs about a politician’s integrity and intelligence; and people’s attitudes toward the death penalty might be based on their beliefs about its fit with personal values. The wealth of important topics pertinent to people’s beliefs may be the reason why scientists who study attitudes have been particularly focused on the role of beliefs (p. 108).

Once again, the deep connection between attitudes and beliefs are shown to be central to research in this area. Additionally, Fishbein and Ajzen (1975) explained that attitude measurement was traditionally done through the measurement of beliefs. Yet, the clear separation within the research here between technology attitudes and teacher technology beliefs is the vital distinction through which each construct can be understood.

Knowledge

Knowledge, specifically knowledge related to technology and technology integration, are two deeply connected and interdependent factors. For this reason, each knowledge component cannot be viewed separately, but must be considered a unified whole. However, each component of technology use should be clarified to understand how it relates to the present report.

Conceptual knowledge is a blend of technological pedagogical knowledge (TPK), technological pedagogical knowledge (TCK), technological pedagogical and content knowledge (TPACK).

Alternately, practical knowledge is simply knowledge about the function of technology, related to technological knowledge (TK). Following is a closer examination of the conception and interrelation of these classifications.

Conceptual knowledge. What is TPACK? TPACK (formally TPCK) is a framework addressing the integration of technology into teaching. Conceptualized by Mishra and Koehler in

2006, Technological Pedagogical Content Knowledge is a widely-used framework within teacher education and professional development, as well as educational research. Figure 1 provides a visual representation of the framework. Mishra and Koehler (2006) view TPACK as the intersection of three unique domains of knowledge (technological, pedagogical, and content knowledge), the hybrid knowledge areas resulting from the intersection of the aforementioned domains (technological content, technological pedagogical, and pedagogical content knowledge), and preeminent intersection now known as TPACK. This framework builds upon the conceptual work of Shulman (1986) addressing pedagogical content knowledge (PCK). PCK encapsulates the intersecting knowledge teachers possess regarding both content areas (e.g., language arts, science, mathematics, etc.) and pedagogy (Shulman, 1986). The next section explores the definition of this type of knowledge.

Figure 2-10. TPACK framework. Reprinted from http://tpack.org.

What is pedagogy and pedagogical content knowledge? In his seminal work on

Pedagogical Content Knowledge (PCK), Shulman (1986) simply defines pedagogy as “how to

teach [content]” (p. 6). This view of pedagogy is deceptively small, and encompasses a wide range of topics within educational theory. Husbands and Pearce (2012) claim effective pedagogy give serious consideration to pupil voice, depends on behavior, understanding, and beliefs, involves long and short term thinking about goals, builds on prior learning and experience, involves scaffolding learning and varying techniques, focuses on dialogues and assessments for learning, and addresses matters of equity. Each of these areas is simultaneously diverse and integrated. Therefore, encapsulating pedagogy as a knowledge domain becomes inherently difficult. For the purposes of this research, it may be necessary to step back from pedagogical knowledge as teacher-centric to a focus on learning theory. Ertmer and Newby (1993) focus on learning theory because learning theories are a source of verified instructional strategies, tactics, and techniques, provide the foundation for intelligent and reasoned strategy selection, and felt integration of the selected strategy within the instructional context is of critical importance.

Regardless of the scope of pedagogical knowledge, the development of such knowledge requires specific questions to be asked. Shulman asked, “What are the sources of teacher knowledge? What does a teacher know and when did he or she come to know it? How is new knowledge acquired, old knowledge retrieved, and both combined to form a new knowledge base?” (1986, p. 8). The answer to these questions provided the conceptualization of Pedagogical

Content Knowledge (PCK). It is interesting to note that Shulman never clarifies pedagogical knowledge in his discourse, but Koehler and Mishra (2009) define this knowledge for Shulman as a teacher’s deep knowledge about the processes and practices or methods of teaching and learning, including: educational purposes, values, and aims. These process and practices separate from content may readily be called pedagogical knowledge (PK). While important to define here, these concepts are purely fundamental to understanding their counterparts involving technology.

The next two sections look at intersections similar to PCK, which are technological pedagogical knowledge (TPK) and technological content knowledge (TCK).

What is technological pedagogical knowledge? When the knowledge domains of technology and pedagogy intersect, it results in the genesis of a new type of knowledge, technological pedagogical knowledge (TPK). Initially, Mishra and Koehler (2006) defined TPK as “knowledge of the existence, components, and capabilities of various technologies as they are used in teaching and learning settings, and conversely, knowing how teaching might change as the result of using particular technologies” (p. 1028). This includes knowing pedagogical affordances and constraints of technologies relating disciplinarily and developmentally appropriate pedagogical designs and strategies (Koehler & Mishra, 2009).

For Pamuk, Ergun, Cakir, Yilmaz, and Ayas (2013), TPK is knowledge about enhancing pedagogical practices with the implementation of technology into teaching and learning activities to enrich or support teaching. Pamuk et al. (2013) state this knowledge represents itself through the implementation of technologies in teaching, enriching pedagogical strategies through authentic experiences, and through development and implementation of alternative assessment strategies (i.e., e-portfolio, blogs, discussion forum). This indicates that TPK requires the ability to adapt and apply both commonplace and emerging technologies.

The challenge to examining TPK is the inherent interconnectedness of the pedagogy, content, and technology. Most commonly researchers look at science education in relation to technology using simulations (Dawley & Dede, 2014; Ma & Nickerson, 2006; Ruben, 1999) or games (Quinn & Connor, 2005). The broader examination of technological pedagogy becomes crucial here because of the increasing exclusive focus on language arts and mathematics in response to standardized testing. Simply focusing on specific content areas only limits the

application of technology across the full range of appropriate PK12 content. To begin, Harris and

Hofer (2009) examine what they describe as a “grounded” approach to technology integration based on content, pedagogy, and how teachers plan instruction. Their method, adopted from a report employs a series of eight continua (lesson center, learning type, prior experience, comprehension level, duration, grouping, and resources) across pedagogical knowledge required for effective instructional design (see Figure 2-11 below).

Figure 2-11. TPK continua. Reprinted from Harris and Hofer (2009).

The need for technology integration within a lesson may come after each of these pedagogical decisions, as suggested by Harris and Hofer, but they may also be seen as potential integration points. As such, TPK can fall at the intersection between the available technological tools, and the affordances of such tools, and the relevant pedagogy to each of the eight continua.

The ever-shifting nature of technology makes TPK extremely difficult to define defined.

The vastness of PK and TK separately makes constraining TPK concretely a nigh impossible.

Additionally, there are some aspects of TK that fall within the area of content, which makes them

potentially irrelevant as a part of TPK. Therefore, TPK would then be defined to contain to the ever-evolving, ever-shifting intersection between learning theory involving technology knowledge and attitudes as they relate to educational purposes, values, and aims.

What is technological content knowledge? Another primary intersection within the

TPACK framework is that of technological content knowledge (TCK). TCK encapsulates knowledge about the manner in which technology and content are reciprocally related (Mishra &

Koehler, 2006). Koehler and Mishra (2009) explain that teachers must deep understanding of the manner in which the subject matter, and the kinds of representations that can be constructed within that domain, can be changed by the application of particular technologies. This understanding means that the teacher knows that a technology changes the nature of learning the subject itself; because new manners of learning exist which were not available previously

(Mishra & Koehler, 2006). Therefore, teachers need to understand which specific technologies are best suited for addressing subject-matter learning in a given domain and how the content dictates or perhaps even changes the technology—or vice versa (Koehler & Mishra, 2009). As such, developing this domain in another key aspect of conceptual knowledge.

Practical knowledge. Practical knowledge of technology can be categorized as technological knowledge (TK)(Koehler & Mishra, 2009; Mishra & Koehler, 2006), Information and Communication Technology (ICT) literacy (National Assessment Governing Board, 2014), or digital literacy (Lankshear & Knobel, 2008). The NAGB found that the terms “technology,”

“information communication technology,” “21st Century skills,” and “literacy” are defined and used in significantly different ways in formal and informal education, in standards, by professional organizations, and in legislation. While each of these terms have been defined earlier throughout this chapter, how do they fit into the concept of practical knowledge about

technology? As previously stated, this practical knowledge may either be connected to the products and processes of technology, or related to skills for the use of information and communications technologies (ICTs). Practical technological knowledge falls into three categories: intellectual capabilities (e.g., sustained reasoning, troubleshooting, collaboration, communication), ICT concepts (e.g., hardware, technological systems, computational thinking), and technology skills (e.g. setting up devices and using basic operating system features, word processors, graphics programs, etc.) (CITL, 1999). Tristán-López and Ylizaliturri-Salcedo (2014) list in detail a wide range of potential technological skills and competences (See Table 2-4).

Additionally, ISTE’s Standards for Teachers emphasized the importance of modeling digital age work and learning (International Society for Technology in Education, 2017d). Measurement of technological knowledge and skills best way to measure technology skills is through complex, real-world performance assessments (Hohlfeld, Ritzhaupt, & Barron, 2010). Examples of such measures for various populations are Student Tool for Technology Literacy (ST2L) (Hohlfeld et al., 2010), the ICT competence scale created by (Aesaert, Van Nijlen, Vanderlinde, & van Braak,

2014), the Inventory of Teacher Technology Skills (Harmes, Barron, & Kemker, 2007), and

ETS’s iSkills Assessment (Katz, 2007).

These views encapsulate technological knowledge, yet there exists an on-going challenge to defining this knowledge. Referring back to the earlier argument, technology is ever evolving and some technologies will disappear, it can be stipulated that much of current technologies possess an evolutionary relationship with the technologies that preceded it (Mishra & Koehler, 2006).

Therefore, TK does not posit an “end state,” but see TK evolving over a lifetime of generative, open-ended interaction with technology (Koehler & Mishra, 2009). Nevertheless, whatever the

current state of technology, there must be some level of practical TK for a teacher to integrate technology, as indicated in the TPACK model.

Table 2-4. Examples of technological knowledge. Adopted from Tristán-López and Ylizaliturri- Salcedo (2014, p. 325). Concept Examples of skills or competencies Concept ICT literacy Access, manage, integrate, evaluate, create, transmit, or communicate information. Navigate in digital environments. Search, locate, retrieve, and selectively sieve sets of data. Classify, organize, analyze, synthesize, store, and creatively produce new information, according to specified formats Install, use, and apply digital or electronic devices, manage data and information. Use of telecommunications and other information-retrieval devices or programs with a computer. It may involve the ability to select how to use specific tools or devices.

Digital literacy Search, locate, organize, analyze, evaluate, and synthesize information to communicate, manage, produce, and perform other tasks with the information. Competencies are related to the management of the information itself rather than the use of any particular device.

Information literacy Search, locate, organize, evaluate, and use information, mainly as an information-retrieval activity where the Internet, databases, microfiches, and other digital documents are the resources needed for this particular application. Competencies are related to the information itself rather than its management or the use of a particular device.

Computer literacy Install, apply, and modify commercial or open-source operating systems and programs. Competencies are related to the ability to use and control the computer and peripherals.

Technology literacy or Understand, select, apply, use, manage, and evaluate all kinds of technology or technological literacy technological products and devices. Synthesize information, draw conclusions about consequences, and make creative interpretations, analyzing data and projecting forecasts or trends. Competencies are related to the use of particular devices to manage information or control specific tasks and data.

ICT advanced and professional Install, upgrade, and configure hardware and software. Verify and optimize the operation of peripheral and communication devices, perform maintenance activities, and update programs to protect the system against viruses and other undesirable software. Support server and network operations, manage and maintain permissions, passwords, and accounts on the various ICT users’ levels.

Measurement of Attitudes, Beliefs, and Knowledge

For this research, it is critical to understand how empirical measurement of attitudes, beliefs, and knowledge has taken place in prior research. While tightly intertwined, the measurement of each technology integration barrier should employ a specialized tool. To that end, researchers have constructed measures specifically for educational context studies (e.g.,

Teo, Lee, and Chai (2008)), or adapted for education from other research areas (e.g., Hudiburg

(1995)). There have also been multi-year longitudinal studies to examine change in features of attitude (e.g., anxiety, enjoyment, avoidance, etc.), beliefs, and skills connected with technology for teachers (Christensen & Knezek, 2001). This study utilized the Teachers’ Attitudes Toward

Information Technology (TAT) questionnaire (Knezek & Christensen, 1998), the Teachers’

Attitudes Toward Computers Questionnaire (TAC version 5.1) (Christensen & Knezek, 1998), the Technology Proficiency Self-Assessment (Ropp, 1999), and four other tool to track changes in attitudes, beliefs, and knowledge over six stages of data collection (Christensen & Knezek,

2001). For reference, Table 2-5 provides a snapshot of the tools either directly applied or adapted for research into teacher (both preservice and inservice) attitudes, beliefs, or knowledge related to technology integration.

Table 2-5. Examples of technology integration attitudes, beliefs, or knowledge measures. Tool Name Focus Developer Description Item Count Technology Skills knowledge Marvin, Lowther, and Ross Assesses perceptions of technology ability as Assessment (TSA) (2002) indicated in the NETS-S grades 6–8 Teachers rate “How easily ...” they can use software features to complete tasks related to computer basics, software basics, 47 multimedia basics, Internet basics, advanced skills, and using technology for learning (Lowther et al., 2008)

Teacher Technology beliefs Unidentified Participants rate their level of agreement Questionnaire (TTQ) with statements regarding five technology- related areas: impact on classroom instruction, impact on students, teacher readiness to integrate technology, overall 20 school support for technology, and technical support. Items are rated on a five-point Likert-type scale (Lowther et al., 2008).

Attitudes Toward Computer attitude/belief Kinzie and Delcourt (1991) Through positively and negatively worded Technologies (ACT) statements measures the perspective on technology’s usefulness (e.g., “Communicating with others over a computer network can help me to be a more 19 effective teacher."), and comfort/anxiety related to technology (e.g., "I feel comfortable about my ability to work with technologies."). Self-Efficacy for Computer attitude Kinzie and Delcourt (1991) SCT measured confidence for word Technologies (SCT) processing, e-mail, and CD-ROM databases.

Table 2-5. Contiued. Tool Name Focus Developer Description Item Count The Computer Hassles Scale attitude Hudiburg (1995) Computer Hassles Scale is a 37-item scale measuring computer users' stress based on the severity of hassles score for the total scale and two subscales, Computer Runtime Errors and Computer Information Problems. 37

The Scales of Attitude beliefs Yavuz (2005) This scale of attitude was developed as a Towards Technology data-collecting tool in order to evaluate the interest and tendencies towards the usage of technological tools by the preservice chemistry teachers. 19

TPACK Survey knowledge Schmidt et al. (2009) This instrument contained items for measuring preservice teachers’ self- assessments of the seven TPACK domains. Each question used a five-level Likert scale. 75

Technology Acceptance attitude Teo et al. (2008) The instrument was adapted from the Measure Technology Acceptance Model (Davis et al., 1989); and was composed of statements measuring perceived useful, perceived ease of use, subjective norm, and facilitating 18 conditions. The measure used a five-point Likert scale.

Table 2-5. Continued. Tool Name Focus Developer Description Item Count Technological Pedagogical attitude Lee and Tsai (2010) Designed to assess teachers’ self-efficacy in Content Knowledge-Web terms of web pedagogical content (TPCK-W) Survey knowledge. The survey was created based on the researchers’ TPCK-W framework including Web knowledge, Web-Content 30 knowledge, Web-Pedagogical knowledge, and Web-Pedagogical-Content knowledge.

General Attitudes Towards attitude van Braak and Goeman GATC scale utilized five-point Likert scale Computers (2003) items related to computer liking, computer anxiety, and computer confidence.

Computer Attitude Scale attitude Loyd and Loyd (1985) A Likert-type instrument measuring four (CAS) categories attitudes on computers and their use. The four categories are: anxiety or fear of computers, confidence of ability to learn or use computers, enjoyment/liking of 40 computers, and perceived usefulness.

Unnamed Survey knowledge Archambault and Crippen To measure online pedagogy, course design, (2006) and technical assistance; this survey asked teachers about their perceptions of their education program and professional development to teach in an online 24 environment. The survey used a 4-point Likert scale.

The measurement tools highlighted above were classified as follows. Those measuring attitude were the tools related to the earlier identified constructs related to attitudes (i.e., computer (technology) anxiety, technophobia, computer (technology) confidence/self-efficacy, technology acceptance, technology adoption, perceived usefulness of technology, and technostress. As seen in the table above, many of these measures employ Likert scales. Maio and

Haddock (2014) explain that Likert scales remain an important tool for researchers interested in assessing attitudes and opinions, so this technique makes sense from a research standpoint.

Belief measurements were either those specifically identified as measuring beliefs, or those which were described as measuring attitude, yet the item statements were more closely to belief statements as defined in this chapter. For example, The Scales of Attitude Towards

Technology, developed by Yavuz (2005), includes these items:

 Technological facilities have a positive effect on productive studying and learning.  Using the Internet in the learning process is a waste of time.  Students should receive basic education on computer literacy (p. 25).

The wording of these examples related more directly to the conceptualization of beliefs for this research. Similarly, the Attitudes Toward Computer Technologies (ACT) (Kinzie & Delcourt,

1991) included items more closely related to beliefs here. However, a majority of the items were attitudinal, so it was grouped as an attitude measure.

The measurement of teacher attitudes, beliefs, and knowledge related to technology integration is not without criticism. Koehler, Shin, and Mishra (2011) explored the existing research regarding TPACK measurement through study-level analysis of empirical studies of

TPACK measures published between 2006 and 2010. Figure 2 summarized the results of the study. The study found there were major gaps in the measurement of TPACK. Most importantly, the study concluded that out of the 141 measurement studies evaluated that nearly 91.5% of the

studies did not provide, or only vaguely provided, validity evidence on the measure (Koehler et al., 2011). Additionally, evidence of reliability was only provided in about 17% of the studies

(Koehler et al., 2011).

Figure 2-12. Findings on TPACK measurements from 2006-2010. Reprinted from Koehler et al. (2011)

Additionally, even validated measure may not always translate well study to study. For example, Ritzhaupt, Huggins-Manley, Ruggles, and Wilson (2016) found that in the TKTT measure by Schmidt et al. (2009) there was lack of clear distinction between the various TPACK subscales. This suggests that measurement of this type of knowledge still has conceptual questions that are yet unaddressed by the measurement literature. This may be why Koehler et al.

(2011) conclude their article by highlighting the benefits of the TPACK framework’s descriptive benefits for measurement, but suggest that the robustness of the research needed improvement, especially qualitative measures. Because of the potential to incorrectly measure attitudes, beliefs, and knowledge as defined in this research, study quality was included as a moderator in the appropriate analyses.

In summary, this research was designed to explore how teacher education courses can influence preservice teachers’ attitudes, beliefs, and knowledge to limit second-order barriers to change. Attitudes are those affective positions related to technology and technology integration.

Beliefs are teacher pedagogical beliefs related to technology integration. Knowledge are the practical and conceptual skills and knowledge needed for effective technology integration. Each of these constructs are measured empirically through tools adapted from other areas or designed specifically for teachers, preservice or inservice, to gauge their ability or stance for each construct. The next section identifies the means in which education can shape attitudes, beliefs, and knowledge; and explain what research says can influence preservice teachers’ views of technology integration.

Changes to Second-Order Barriers Through Education

In addition to understanding the impact of teacher education courses on technology integration attitudes, beliefs, and knowledge; this research was deigned to explore the impact of specific course design elements. Theory shows that changes in attitudes, beliefs, and knowledge require special features. This section looks at how attitudes, beliefs, and knowledge are formed and changed; as well as exploring those features of technology integration courses that research shows influence technology integration.

Changes in attitudes

Attitudes, as theorized in this research, are related to the affective positioning of the individual. Outcomes in the affective domain relate to the emotional abilities desired for students

(Krathwohl, Bloom, & Masia, 1964). Krathwohl et al. (1964) identified five levels of affective outcomes: receiving, responding, valuing, organizing, and those characterized by a value complex. Valuing and organizing are the affective outcomes most relevant to the current research. Morrison et al. (2013) explain that valuing indicated a willingness to accept or reject an

event through the expression of an attitude. Organizing means that when a student encounters situations to which more than one value applies, he/she is willing to organize the values, determine relationships among values, and accept some values as dominant over others

(Morrison et al., 2013). The levels of the affective domain form a continuum for attitudinal behavior, from simple awareness and acceptance to internalization, as technology integration attitudes become part of an individual’s practice (Morrison et al., 2013). In each case, the value goal of the teacher education course would be for the preservice teacher to prioritize technology integration. The outcomes of the course would manifest through the student speaking to the value of technology integration, choosing to integrate technology, and integrating technology.

Changes in attitude can come through a variety of techniques. Fishbein and Ajzen (1975) present Skinner’s operant conditioning and other learning theories of (e.g., the congruity principle Osgood, Suci, and Tannenbaum (1957)) as impactful for modifying the behavior resultant from the attitude. Additional research identifies more recent models (e.g., Flexible

Correction Model, Affect Infusion Model, etc.) as working for shaping attitudes (Forgas, 2008).

As the specific attitudinal shift mechanism is outside the scope of this research, the research cited here serve only to provide examples of how attitude change is addressed in research.

Changes in beliefs

Beliefs are teacher pedagogical beliefs related to technology integration. Pajares (1992) suggests that belief change for adults is a rare phenomenon, the most common cause being a conversion from one authority to another or a gestalt shift. He goes on to say, “Individuals tend to hold on to beliefs based on incorrect or incomplete knowledge, even after scientifically correct explanations are presented to them” (Pajares, 1992, p. 325). A change in belief may result from effectively reaching the highest level of affective change (i.e., a value complex). Morrison et al.

(2013) explain that a value complex means the individual consistently acts in accordance with

accepted values and incorporating that behavior as a part of one’s personality (p. 105).

Therefore, the result of the technology integration course results in the preservice teacher adopting a technology integration value complex.

Changes to teacher pedagogical beliefs could present themselves in any number of ways.

Marsh and Wallace (2005) summarize the research on the connection between attitude change and belief change, which suggests that beliefs will change as the result of a shift in attitude. They identify thought-induced attitude polarization (i.e., visualizing a positive or negative association) and thought introspection (i.e., reflection on a belief’s root cause) can both initiate belief change

(Marsh & Wallace, 2005). However, some research also suggests that belief change can proceed attitude change (Marsh & Wallace, 2005). Therefore, course features that promote a positive technology integration attitude may impact the belief structure. Either stance works equally well within this study.

Changes in knowledge

Knowledge has been defined here as the practical and conceptual skills and knowledge needed for effective technology integration. Practical knowledge of technologies are the skills and knowledge specifically related to technological knowledge (TK) or digital literacy.

Conceptual knowledge are the skills and knowledge related to both technological pedagogical knowledge, and, to some degree, technological pedagogical content knowledge (TPACK). In revising Bloom’s Taxonomy, Anderson, Krathwohl, and Bloom (2001) categorize four dimensions of knowledge: factual, conceptual, procedural, and metacognitive. Alternately, Niess

(2008) uses the following descriptions:

declarative (knowing that, including definitions, terms, facts, and descriptions), procedural (knowing how that refers to sequences of steps to complete a task or subtask), schematic (knowing why by drawing on both declarative and procedural knowledge, such as principles and mental models), and strategic (knowing when

and where to use domain-specific knowledge and strategies, such as planning and problem solving together with monitoring progress towards a goal) (p. 256).

Each of these types knowledge can apply to both the practical or conceptual knowledge defined in this chapter. Morrison et al. (2013) suggest that too often major attention is given to the lowest cognitive level, or memorizing and recalling. However, technology integration is a highly inter- domain practice, which suggests that focusing on higher-order thinking will be crucial for effective teacher education.

Understanding change in knowledge is a vast domain. Various principles (e.g., Merrill’s

(2002) First Principles of Instruction), theories (e.g., Gagné’s (1970) conditions of learning,

Mayer’s (2005) cognitive theory of multimedia learning etc.), and design elements (e.g., van

Merriënboer and Kirschner’s (2012) four-component instructional design) all become relevant when discussing cognitive learning. Nilson (2016) and Morrison et al. (2013) highlight several features relevant for adult learners, such as authentic learning, socially constructed knowledge, and student-centered learning. While defining and conceptualizing the full scope of education is outside the scope of this dissertation, identifying key structures for advancing technology integration in preservice teachers becomes imperative. There, the next section is devoted to a review of the literature related to technology integration to identify significant features of technology integration courses for promoting technology integration.

Identified Features of Teacher Education Courses for Technology Integration

Teacher education programs have been distinguished in the literature by a variety of structural terms, such as: length, when they are offered (i.e., undergraduate vs. graduate), sponsoring institution (e.g., college, university, school district, etc.), admissions requirements, and/or curricular focus (Zeichner, 2005). The research at hand focuses on university teacher education programs that have stand-alone technology integration courses as part of a four- or

five-year program. Some research (e.g., Koehler and Mishra (2005); Park, Gilbreath, Lawson, and Williams (2010)) suggest that technology integration should be embedded throughout the program courses. However, this research will not explore such programs to better constrain the parameters of the research. Nevertheless, for these courses some features are especially relevant for shaping attitudes, beliefs, and knowledge related to technology integration. A review of the literature identified eight course features in technology integration courses: mentoring/coaching, rehearsal/field experience, goal-setting, observation, reflection/self-evaluation, hands-on learning, work sample analysis, and practice lesson planning. Some of these features were identified through the literature review process, and others by the systematic review of the literature. The benefits of these features were not necessarily investigated as part of the research, but were course components. This section explores each of these features.

Mentoring/Coaching

The first identified feature that supports technology integration education is mentoring or coaching by a technology integrating teacher (Goktas et al., 2009; Kay, 2006). Mentoring or coaching means the active encouragement and support of a technology integrating teacher. In

Kay’s literature review, the collaborative relationship between the preservice and technology integrating mentor teacher working together to produce meaningful technology use was shown in the research to have a positive effect (Kay, 2006). Kay (2006) explained that this benefits from taking less time than the full-collaborative model involving partnerships among faculty, mentor teachers, and preservice candidates. In another study, Lowther et al. (2008) found that when a technology coach provided one-on-one support and encouragement by showing benefits of technology that there were changes in teacher beliefs. Of course, it is important to note that how well the mentor teacher contributes to the student teacher’s education dependent on their technology integration attitudes, knowledge, and beliefs (Enochsson & Rizza, 2009). The use of

mentors or coaches relates directly to the benefits of collaboration that Ertmer (1999) described as valuable to eliminating barriers. Ertmer explained the social-cultural interactions that through ongoing conversations, engagement in technology projects, and shared planning time, teachers access a supportive network that empowers them first to envision, and then to achieve, meaningful technology use (1999, 2005). A review by Chuang, Thompson, and Schmidt (2003) found that mentorship models promoted collaboration for technology use, a reciprocal benefit, and vision for technology use.

Rehearsal/Field-Experience

“If beliefs are formed through personal experience, then changes in beliefs might also be facilitated through experience” (Ertmer, 2005, p. 32). Rehearsal or field experience involves affording preservice teachers the opportunity to plan and implement technology integrated lessons with actual PK12 students. Koehler and Mishra (2005) argue that teacher change cannot be achieved merely through direct instruction, but requires teachers to experience, as learners, the kinds of novel learning environments that can facilitate and enhance learning through the appropriate use of technology. Kay (2006) and Pope, Hare, and Howardy (2002) support the view that rehearsal by preservice teachers learn from designing and implementing hands-on, technology-integrated learning experiences for classroom students allows them to focus on how technology affects learning in the classroom. Yet, field experiences are not without issues.

Additionally, research has shown (Enochsson & Rizza, 2009; Strudler & Wetzel, 1999) that access can be an issue for some preservice teachers limiting the effectiveness of this practice.

Nevertheless, presuming the absence of access issues, field experiences have shown to be helpful in positively influencing technology integration practice.

Goal-Setting

The next feature that promotes technology integration practice education through teacher education courses is goal-setting. This means the setting of specific goals for the improvement of technology integration practice by the preservice teacher. These goals should focus on students’ needs, especially when making classroom implementation decisions (Ertmer, Ottenbreit-

Leftwich, & York, 2006). The benefit of setting goals lies in the establishment of specific performance levels with concrete, measurable outcomes (L. Wang et al., 2004). Ertmer and

Ottenbreit-Leftwich (2010) suggest goals may include meeting regularly to monitor progress or encouraging self-assessment. The concept of goal-setting coincides with ISTE’s conditions of a shared vision and ongoing professional learning (International Society for Technology in

Education, 2017e). Studies found preservice teachers who used specific goals experienced significantly greater increases in attitude change (L. Wang et al., 2004). When coupled with vicarious learning experiences, L. Wang et al. (2004) found preservice teachers experienced greater increases in judgments of computer self-efficacy than those who received only one of these two conditions. There may even be a synergistic effect as some teachers find technology allows them to achieve their current goals more effectively than do their traditional methods

(Zhao & Cziko, 2001). As such, an interrelated goal of improved technology use through technology use could lead to greater gains.

Observation

Observation has been found to be a powerful component of teacher education courses related to technology integration (Ertmer, 2005; L. Wang et al., 2004). Observation, also called vicarious experiences, has been shown to influence teacher attitude (Ertmer, 2005). Observation means the preservice teacher can watch an inservice teacher effectively integrate technology in the classroom. These observations could be with the preservice teacher’s mentor, or with any

exemplary teacher. This differs from mentoring (discussed later) in that observations do not necessarily involve additional coaching on the part of the inservice teacher. The observation of peers, mentors, or seasoned practitioners can illustrate effective ways to use technology to teach

(Ertmer, 1999). Research by Wang et al. (2004) showed that observing teachers using computers during the student teaching was one of the three most important factors that influenced feelings of preparedness for the use of computers for instruction in their own classrooms. Additionally,

Lim and Khine (2006) found a “buddy system,” a merging of mentorship and observation, for observation yielded positive results.

Reflection/Self-Evaluation

Reflection or self-evaluation was identified in the literature as a component that may support the development of technology integration practice in teachers. Rodgers (2002) explains that this practice is rigorous and systematic and distinct from less-structured kinds of thinking that mirrors inquiry or scientific process. Koh and Divaharan (2011) and Angeli (2005) found that reflection on technology integration practice can facilitate the development of technological and technology integration knowledge. Bai and Ertmer (2008) suggest that when beliefs are subjected to deeper reflection they can be changed over time, but typically after being challenged by new information or opposing beliefs. Ertmer and Ottenbreit-Leftwich (2010) advocate for the use of reflection in the development of each attitudes, beliefs, and knowledge of technology integration. Mouza, Karchmer-Klein, Nandakumar, Ozden, and Hu (2014) contend that metacognitive reflection allows preservice teachers to consider technology, content, and pedagogy and their inter-relationship when considering specific instructional problems. These and other examples suggest that reflection or self-evaluation may be beneficial for teacher education in this area.

Hands-On Learning

Hands-on learning (i.e., experiential learning) is a method of learning whereby student learn through a process of experiences that holistically allows the learner to interact with their environment (e.g., technologies) to create knowledge (Kolb & Kolb, 2005). This process can out the students' beliefs and ideas about a topic so that they can be examined, tested, and integrated with new, more refined ideas (Kolb & Kolb, 2005). Hew and Brush (2007) highlight hands-on learning as a key component of effective programs fostering technology integration in teachers.

Koehler, Mishra, and Yahya (2007) recommend learning-by-doing and problem-based learning to support the development of TPACK specifically. They found this method leads to sustained inquiry and revision of thinking. Koh and Divaharan (2011) report schools of education that use hands-on projects were effective for helping preservice teachers to foster technology integration know-how. Vannatta and Beyerbach (2000) show that hand-on learning resulted in a change of attitudes and beliefs about the practical application for teaching and learning.

Work Sample Analysis

In this study, work sample analysis consists of any course activity that critiqued or reviewed any practitioner created materials (e.g., lesson plans) or enacted lessons (e.g., via video analyses) involved technology integrated lessons. This may have involved the feature being described as work sample analysis, video analysis, or lesson critique. Mouza et al. (2014) found the addition of this course feature requires preservice teachers to think deeply about content, pedagogy and technology as well as how these three constructs combine to develop effective instruction, and thereby lead to gains in knowledge for technology integration. Such analyses may also make changes to attitudes and beliefs. As discussed earlier, reflection can benefit the process of developing technology integration beliefs and attitudes. Work sample analysis can be

considered a version of reflection, but instead the focus is on a benchmark created by professional educators rather than the preservice teacher’s own work and practice.

Practice Lesson Planning

Practice lesson planning is a commonly used tool of teacher education courses. For example, Ottenbreit-Leftwich, Glazewski, Newby, and Ertmer (2010), Brush et al. (2001),

Koehler et al. (2007) each identified practice lesson planning as a course feature to influence either attitudes, beliefs, and especially knowledge about the practice of designing effective educational experiences using technology. Designing lessons as part of coursework allows the preservice teacher to get feedback on how they have conceptualized and operationalized technology within a lesson. These lesson plans were either specific course assignment, or part of a larger series of tasks that involved them implementing a technology integrated lesson and reflecting on their practice.

Conclusion

Technology integration is a vital component of modern education. However, there are barriers to technology integration by teachers. The second-order barriers to technology integration may possibly be mitigated or eliminated by teacher preparation courses. Yet, the extent to which these courses on technology integration is not fully known. Therefore, the research here proposes a meta-analysis of studies of change in attitudes, beliefs, and knowledge though technology integration courses to quantify that effect. The courses under examination are those in teacher education at universities with four- or five-year programs and stand-alone technology integration courses. The study also examined the impact of eight course features: mentoring/coaching, rehearsal/field experience, goal-setting, observation, reflection/self- evaluation, hands-on learning, work sample analysis, and practice lesson planning.

CHAPTER 3 METHODS

Introduction

This study utilized a systematic review process and meta-analyses techniques to evaluate the effect of teacher education courses for technology integration (TECTI) on the attitudes, beliefs, and knowledge (i.e., second-order barriers constructs) of preservice teachers training to become PK12 educators. The courses under examination are those in teacher education at universities with four- or five-year programs that act as stand-alone technology integration courses (either singularly or part of a series). This research sought to quantify what, if any, positive change in attitudes, beliefs, or knowledge potentially result from the courses. Understanding this change could help researchers, teacher educators, and policymakers better understand the influence of stand-alone courses so that teacher education targeted at technology integration might better prepare future teachers. Furthermore, the sub-group meta-analyses sought to quantify the impact moderating features of course feature, overall study quality, and measurement tool validity. These sub-group analyses both aid in empirically informing both instructional and research design in this domain. Specifically, systematic review and meta-analyses were applied to explore the following question: What is the effect of TECTI on second- order barrier constructs (i.e., teacher attitudes, beliefs, and knowledge)? Furthermore, the research investigated a series of subgroup analyses to answer:

 What is the difference in magnitude of effect of TECTI on each second-order barrier construct when comparing levels of overall study quality?

 What is the difference in magnitude of effect of TECTI on each second-order barrier when comparing levels of measure validity?

 What is the difference in magnitude of effect of TECTI on each second-order barrier when comparing levels of reported reliability?

A series of meta-analyses into the impact of teacher education courses for technology integration (TECTI) on attitudes, beliefs, and knowledge was selected for a variety of reasons. As hypothesized earlier, TECTI have the potential to impact attitudes, beliefs, and knowledge of preservice teachers both during study and in their future practice. Furthermore, TECTI have been shown to have potential for mitigating or eliminating second-order barriers for technology integration. Regarding the first stage of the research, systematic review, Pigott (2012) states that a systematic review promise a transparent and replicable method for summarizing the literature to improve both policy decisions, and the design of new studies. The systematic review for this research examined over 3000 research articles across multiple databases and other repositories. As such, this research collected valuable primary studies across a decade of research to help inform the results. The subsequent meta-analyses explore the consistency of effect across the identified studies (Borenstein et al., 2009). Furthermore, if there is a lack of consistency, meta-analyses can quantify the extent of such variance (Borenstein et al.,

2009). A deeper understanding of the quantifiable effect of the TECTI on preservice teachers’ attitudes, beliefs, and knowledge allows for multiple benefits. First, this research will define a quantifiable measurement of the impact TECTI have on the outcome variables. Next, as a direct result of the quantified impact, the subsequent sub- analyses will provide a clearer vision of the specific course features in TECTI that best impact the outcomes, and how study design and both measure validity and reliability impact the estimation of average effect size.

The rest of this chapter details the methods of systematic review and meta- analysis. First, an overview of meta-analysis is given, including the benefits and critiques of the method. Second, a look at recommendations regarding the execution of systematic review and meta-analysis is detailed. Finally, a description of the methods applied to this current research study is detailed in full.

Meta-Analysis Overview

Since the introduction of meta-analysis, its use in combining results of replicated research studies has become widespread in education, psychology, and the biomedical sciences (Borenstein et al., 2009). As with all methods, meta-analysis is not without its strengths and weaknesses, and as such there are important considerations to be made when conducting a meta-analysis. Borenstein et al. (2009) argue that opposed to a narrative exploration of research, meta-analysis address issues in research that help contextualize results of studies in light of other research. Wolf (1986) echoes the view of

Borenstein et al., and more fully explains that traditional literature reviews suffer from selective inclusion of studies, differential subjective weighting and misleading interpretation of studies, failure to examine study traits for the underlying sources of varying results, and failure to examine moderating variables in the relationships under examination. In contrast, meta-analysis has been viewed as an efficient way to summarize large literature pools, while providing several distinct advantages over traditional methods of synthesis (Cooper, 2017; Liao & Bright, 1993). These advantages include a systematic, clearly articulated, and replicable approach to integrating findings; the aggregate effects of studies enlarges sample sizes and increases power, patterns of relationships can become apparent and the probability of detecting an effect increase, and paint a comprehensive picture of results over a large range of varying methods to name a

few. Additionally, Borenstein et al. (2009) contend researchers need to know if the effect size of a study is consistent across research, and if it is estimate the effect size accurately in a manner that is robust across the literature in a synthesis. The calculation of the effect size allows researchers to understand the magnitude of the effect of a treatment or intervention, unlike a single study, where the p-value only indicates if the result of the intervention is statistically different from the null hypothesis (Borenstein et al., 2009;

Ioannidis & Lau, 1999). In all, meta-analysis allows researchers to clarify the effect of an intervention, or in other words examine the consistency of the effect of an intervention/treatment (Borenstein et al., 2009). The overarching benefit of this clarification is that meta-analysis becomes useful for highlighting literature gaps, providing new research direction, and finding mediating of interactional relationships too subtle to be observed or hypothesized and tested in a single study (Liao & Bright, 1993).

As with other research methods, meta-analysis has its fair share of criticism. Liao and Bright (1993) groups the major critiques of meta-analysis into four categories:

1. Logical conclusions cannot be drawn by comparing and aggregating studies that include different measuring techniques, definitions of variables (e.g., treatments, outcomes) and subjects because they are too dissimilar.

2. The inclusion of methodologically poor studies in the review can result in uninterpretable results in meta-analysis.

3. The representation of individual studies by multiple effect sizes can result in nonindependent data points, and misleadingly large samples.

4. The selection bias in reported research can lead to biased meta-analysis results (p. 96).

Each of these specific criticisms deserves a more in depth description. There are several arguments to be made regarding the ability of meta-analysis to aggregate studies.

The common criticism is that heterogeneity is unavoidable (Ioannidis & Lau, 1999), and as such the summary effect ignored important differences in studies (Borenstein et al.,

2009). However, Borenstein et al. (2009) counter-argue that the purpose of a meta- analysis is to look at broader research questions, so instead of looking at the differences between apples and oranges the researcher is looking at fruit. Not dissimilar to the first critique, the second main concern about meta-analysis is the inclusion of methodologically poor studies. It is true that meta-analyses may end up containing garbage studies, and thus the results analysis may too be considered garbage. Yet, meta- analysis cannot improve the quality of earlier studies (Ioannidis & Lau, 1999). While reasonable to illuminate concerns about the quality of included studies, it is not entirely fair to fault the methodology. Any systematic review has inclusion and exclusion criteria, and it becomes the responsibility of the researcher to control the quality of the studies included (Borenstein et al., 2009). The third challenge to meta-analysis as a method comes from the misleading effects of the method. Ioannidis and Lau (1999) argue meta- analysis is an effective counter for this issue because of its ability to detect the effect of smaller, statistically non-significant results in comparison with other trials based on scale.

Wolf (1986) proposes weighting studies as a counter to this problem. The selection bias denunciation has a several different layers. First, publication bias results in negative trials of statistically insignificant potentially have a more difficult time getting published

(Ioannidis & Lau, 1999). Second, non-English speaking literature may be left out when not statistically significant, aka Tower of Babel bias (Ioannidis & Lau, 1999). Third, some studies that may have important impact on a meta-analysis may be filed away, never to get published at all (Borenstein et al., 2009). Like the problem with the inclusion of methodologically poor studies, the problem of selection bias can be negated with appropriate inclusion/exclusion criteria (Borenstein et al., 2009). Furthermore, plotting and visually interpreting a funnel plot of effect size by sample size or variance is another

option to limit the effect of publication bias on the meta-analysis (Borenstein et al.,

2009). One of the biggest arguments against meta-analysis is that the method is enacted poorly (Borenstein et al., 2009). While this could surely be said for any method, the correct application of meta-analysis, as with all research methodology, requires the diligent commitment to sound practice.

Meta-analysis, like all research methods, is not without both benefits and shortcomings. However, properly executed systematic reviews and meta-analyses can provide valuable examinations of the literature and empirical results. The next section examines recommendations for the execution of each of these methodological processes.

Recommendations for Systematic Review and Meta-Analysis

To structure a sound systematic review and meta-analysis, Cooper (2017) outlines a seven-step approach for conducting such research. The stages and subsequent sub-steps are outlined in Table 3-1 below. The following sections examine each of these steps in greater detail.

Table 3-1. Outline of the meta-analytic process from Cooper (2017). Stage Name Stage Components Step 1: Formulating the problem  Defining key study variables.  Corresponding concepts and operations.  Defining the relationship of interest. Step 2: Searching the literature  Database searching.  Identifying research-to-researcher contacts.  Identifying quality controlled channels.  Identifying secondary channels. Step 3: Gathering information from studies  Defining inclusion/exclusion criteria.  Developing a coding guide.  Selecting and training coders.  Identifying problems in gathered data.  Identifying independent comparisons. Step 4: Evaluating the quality of studies  Addressing issues in evaluating study quality.  Defining quality assessment approach.  Identifying statistical outliers. Step 5: Analyzing and integrating the  Analyzing main and interaction effects. outcome of studies  Measuring relationship strength.  Combining study effect sizes.  Analyzing variance across studies.

Table 3-1. Continued. Stage Name Stage Components Step 6: Interpreting the evidence  Addressing missing data.  Statistical sensitivity analysis.  Interpretation of effect size. Step 7: Presenting the results  Report writing in social research.  Meta-analysis reporting standards (MARS).

Step 1: Formulating the Problem

In formulating a meta-analysis’ focus, Cooper (2017) advises answering three specific questions:

 Should the results of the research be expressed in numbers or narrative?

 Is the problem you are studying a description of an event, an association between events, or a causal explanation of an event?

 Does the problem or hypothesis seek to understand (a) how a process unfolds within an individual participant over time, or (b) what is associated with or explains variation between participants or groups of participants (p. 42)?

The relationship of these factors is illustrated in Figure 3-1. The complex nature of the differences that arise in research due to how concepts are defined, operationalized, and relate to one another requires researchers to explicitly define the relevant concepts under consideration (Cooper, 2017).

Question 1 is a particularly misleading question in that both quantitative and qualitative outcomes needed reporting to effectively address the problem, a fact which is acknowledged by (Cooper, 2017). Quantitative questions were formulated from the qualitative research synthesis. Furthermore, qualitative processes aided in the identification of potentially important sub-group and moderator analyses to be conducted.

Question 2 required the consideration of the nature of the relationship to be examined. Descriptive research explores a general “What is happening?” question

(Cooper, 2017). Associative research looks more at the connection between two or more events (Cooper, 2017). Causal research allows researchers to isolate and draw a direct

productive link between a cause and effect (Cooper, 2017). Each of these can be the appropriate outcome of a meta-analysis depending on the nature of the methodological design.

Figure 3-1. Conceptualization of meta-analytic research. Adapted from Cooper (2017).

Finally, Question 3 is a question of participant focus. Cooper (2017) explains that the difference lies in if the research looks at an individual’s change (within-participant) or the association or variance between participant or groups of participants (between- participant). Both are again appropriate for meta-analyses, but are clearly connected to specific methodological choices.

Step 2: Searching the Literature

Searching the available literature for potentially viable studies for inclusion in a meta-analysis involves a detail systematic review of the literature available. Cooper

(2017) identifies a number of potential sources of research studies for meta-analysis.

Database searches, Internet searches, or personal contacts are just some of the means for

obtaining potential articles (Cooper, 2017). To extend such methods, Cooper (2017) further recommends citation/bibliography searches or forward reference searches.

One major concern within meta-analyses is the potential issue of publication bias.

Rothstein, Sutton, and Borenstein (2006) explain that when the readily available research differs greatly from all research done in an area, then the wrong conclusions can be drawn. As a result, publication bias may be the creates threat to the validity of a meta- analysis (Rothstein et al., 2006). Missing data in meta-analysis can impact the results of the process by imposing bias on the results and thereby influencing the conclusions drawn from the research (Cooper, 2017). To some extent, missing data can be avoided by taking aims to avoid the “file drawer problem” (Rosenthal, 1979), which results from non-publishing of statistically non-significant results. The attempts to gather unpublished research from experts in the field detailed earlier this chapter were the first step in avoiding potential bias from missing data. However, this still could not present a full picture of the topic at hand. Therefore, measures were employed to calculate the impact of bias. This bias could have run from trivial (i.e., “If all relevant studies were included effect size would probably remain largely unchanged.”) to substantial (i.e., “If all relevant studies were included, the key finding could change.”) impact (Borenstein et al., 2009, p.

288). To estimate the potential effect of bias, Duval and Tweedie (2000a, 2000b) advocate for the application of a trim-and-fill funnel plot. This procedure uses an iterative process to remove extreme small studies, and re-computing the effect size with a symmetric funnel (Borenstein et al., 2009). These graphical tools plot the potentially missing studies, and require a subjective interpretation by the researcher. Conversely, potential publication bias can be quantified through the calculation of a Fail-safe N

(Borenstein et al., 2009). First proposed by Rosenthal (1979), he provided a method for

100

the statistical estimation of the number of missing studies needed for retrieval and incorporation into the study before the p-value become nonsignificant. Fail-safe N is calculated to aid in the evaluation of potential publication bias using the formula

2 [∑ 푍(푝푖)] 푁푅 = 2 − 푛 (3-1) 푍훼 where n is the number of studies, Z(pi) are the Z-scores for the individual significance values, and Z is the one-tailed Z-score associated with the desired α (Rosenberg, 2005). A second method of estimating the Fail-safe N was proposed by Orwin in 1983 that extended the method to estimate N using standardized effect size as the metric (Becker,

2005), using the formula

푘(푑̅푂−푑퐶) 푁푒푠 = (3-2) 푑퐶−푑̅퐹푆 asking how many effect sizes averaging a particular value (dO) to a particular criterion level (dC). Unlike Rosenthal’s method, Orwin’s method does not require a reduction to non-significance (Becker, 2005). A final method for estimating N was proposed by

Rosenberg (2005) that addressed multiple issues he identified in the previous two methods, including the practice of weighting effect sizes. Rosenberg’s new method provided a general, weighted fail-safe calculation grounded in the meta-analysis framework for both fixed-effect and random-effects models (Rosenberg, 2005). Each of the three methods are commonly used in practice today.

Step 3: Gathering Information from Studies

Cooper (2017) defines five key features for gathering information from identified studies: defining inclusion/exclusion criteria, developing a coding guide, selecting and training coders, identifying problems in the gathered data, and identifying independent comparisons. In this section, each stage of this process is defined.

101

The function of inclusion/exclusion criteria is to tie conceptual variables to observable research operations and measurements (Cooper, 2017). Cooper, Hedges, and

Valentine (2009) recommend setting the eligibility criteria for a meta-analysis under four categories: type of participants, types of interventions, types of outcome measures, and types of studies. Coming to a clear understanding of these features helps conceptualize the research.

Developing a coding guide requires identification of the particular aspects of interest that may appear across studies (Cooper, 2017). Cooper (2017) recommends assigning each coding question topic a letter identifier (e.g., intervention (I), setting (S), participant (P), etc.). The process coding must demonstrate data extraction for the study addresses the issues of bias, idiosyncratic interpretation of coding questions, and/or mechanical error in the process, which are often amplified by single coders (Cooper,

2017). To avoid such issues, qualified coders for this meta-analysis are necessary features of the process. Cooper (2017) recommends pilot testing the codebook with the coders to allow for greater clarity in questions, responses, and conventions of the coding process.

The coder recruit recommended changes to the protocol, which are adapted to the code sheet and book. Having finalized the coding procedure, each coder can codify the gathered research using the defined code protocol. Cooper (2017) proposes two methods to estimate reliability in the coding process. First, the agreement rate (i.e., the number of agreed-on codes divided by the total number of coding opportunities) between pairs of coders can be reported (Cooper, 2017). Alternately, Cohen’s kappa (i.e., the improvement over chance reached by the coders) can be used (Cooper, 2017). Cohen (1960) established the computation of kappa with the following formula:

102

푝 − 푝 휅 = 표 푐 (3-3) 1 − 푝푐

For kappa, po represents the proportion of units in which the judges agreed, and pc represents the proportion of units for which agreement is expected by chance.

There are multiple issues that can cause problems with the gathered data. Cooper

(2017) identifies incomplete reporting of statistical outcomes and incomplete reporting of other study characteristics as primary issues related to meta-analysis related to study retrieval. Additionally, Cooper (2017) explains that often article derived from dissertations will provide the requisite information in the study and/or appendices. When effect size data is unavailable, Cooper (2017) states that a null result presumes an outcome of equal means. This method is typically applied in modern meta-analyses, because its application changes the distribution of the findings.

Effect size calculations

Calculating effect sizes within identified studies depended on the initial study design. One option is an experimental or quasi-experimental design employing one control group and one or more treatment groups. Optionally, researchers may have conducted pre-/post-test analyses or correlational analyses on data. In the case of experimental or quasi-experimental designs, Dunst, Hamby, and Trivette (2004) advise six possible formulas based on the available statistical information. A full list of possible formulas is provided in Appendix A. Most commonly, the formula for comparison of means (M) with available standard deviations (SD) is:

(푀 − 푀 ) 푑 = 퐸 퐶 ⁄ ( 2 2) (3-4) √ 푆퐷퐸 + 푆퐷퐶 ⁄ 2

103

When a pre-/post-test design was used for the original study, a nonindependent sample design method for the calculation of Cohen’s d will be required. Dunst et al. (2004) recommend the following formula when means are provided:

(푀 − 푀 ) 푑 = 2 1 ( 2 2) (3-5) √ 푆퐷1 + 푆퐷2 ⁄ 2

When researchers employ a t-test, Dunst et al. (2004) recommend this alternate formula:

√2(1 − 푟) (3-6) 푑 = 푡 ⁄푁

These are a couple examples of how effect sizes are calculated dependent on study design and available data. Formulas appropriate to those considerations should be applied as needed.

Because Cohen’s d has a slight bias, which causes it to overestimate the absolute value of the effect size parameter (δ) based on sample size, Hedges’ g is applied as a correction (Borenstein et al., 2009). Hedges (1981) coverts d to g using a correlation factor (J):

3 퐽 = 1 − (3-7) 4푑푓 − 1

The correction is applied to Cohen’s d, as well as its variance and standard error

(Borenstein et al., 2009):

푔 (3-8) = 퐽 × 푑

104

Occasionally, studies report multiple measure that can be used to calculate effect sizes. Multiple outcomes or time-points within a study if treated as separate effects sizes creates a violation of the assumption of independence (Borenstein et al., 2009). To mitigate this violation, Borenstein et al. (2009) proposes using robust variance estimation

(RVE). Their formula compensates for the interdependence of the effects sizes and variance.

There are multiple issues that can cause problems with the gathered data. Cooper

(2017) identifies incomplete reporting of statistical outcomes and incomplete reporting of other study characteristics as primary issues related to meta-analysis related to study retrieval. To address shortcomings in potentially fruitful research, one of four methodological steps can be applied in meta-analysis. First, if statistical outcomes are underreported (e.g., missing means and standard deviations, missing sample sizes, etc.) a researcher can look for the data in other sources (e.g., websites connected to the study context). A second option would be to exclude the study. Finally, a researcher could attempt to impute the data from contextual clues of the report (e.g., assume an equal gender distribution). While these methods are not ideal, they are standard to the practice of meta-analysis.

Step 4: Evaluating Study Quality

Study quality is evaluated in one of two ways, but each presents a unique problem in application. A priori exclusion involves individual, objective assessment of a study based on with the goal of reducing bias by identifying, appraising, and synthesizing all relevant studies on a particular topic (Uman, 2011). Unfortunately, such exclusions are subject to personal biases varying from judge-to-judge, thereby being a matter of opinion 105

(Cooper, 2017). Conversely, quality judgments may be made a posteriori using an empirical evaluation of study quality (Cooper, 2017). However, these quality control measures may allow for the inclusion of poor quality studies to start, as well as being comprised of a scale of quantitative measures that are not comparable (Cooper, 2017). As such, Cooper (2017) argues for a blending of a priori and a posteriori quality assessments.

Step 5: Analyzing and Integrating the Outcome of Studies

Main effect estimation

The main goal of any meta-analysis is to use observed effects to estimate a population effect, which requires the collection of Yi to estimate the overall mean

(Borenstein et al., 2009). Doing so required calculation of the within-study variance and

τ2. Borenstein et al. (2009) expresses tau as:

푄 − 푑푓 푇2 = (3-9) 퐶

where

푘 2 (∑푘 푊 푌 ) 푄 = ∑ 푊 푌2 − 푖=1 푖 푖 (3-10) 푖 푖 ∑푘 푊 푖=1 푖=1 푖

and

푑푓 = 푘 − 1 (3-11)

where k is the number of studies, and

2 ∑ 푊푖 퐶 = ∑ 푊푖 − (3-12) ∑ 푊푖

(pp. 72-73).

106

There are two potential models within a meta-analysis: fixed-effect or random- effects. A fixed-effect model compares the variation in observed effect sizes making the assumption that there is one effect size value underlying all observations (Cooper, 2017).

A random-effects model assumes that there are factors influencing each study in a meta- analysis that are not constant across said studies. Borenstein et al. (2009) explain that the random-effects model is best applied when the effect sizes under consideration are expected to be similar, but not identical, across studies because of variation in the participants. In cases where study uniformity is in question, a random-effects model is recommended. Using a random-effects model the assumption is that the true effect size of each study varies between studies, and that the studies identified as part of the meta- analysis are a random sample of effect sizes (Borenstein et al., 2009).

Sub-group analyses

For sub-group analyses, certain features of the research impact design selections.

Borenstein et al. (2009) mapped the following flowchart for sub-group analysis:

Figure 3-2. Sub-group analysis flowchart. Adapted from Borenstein et al. (2009).

The flow chart aids in the selection of an appropriate model for these analyses.

107

Heterogeneity analysis

When a meta-analysis is conducted, Cooper (2017) suggests that it is always import to ask, “Was the homogeneity of effect sizes tested” (p. 241)? Quantifying the homogeneity, or its opposite, heterogeneity, is vital to understanding the results of meta- analyses, because the dispersion in observed effects is partly spurious (Borenstein et al.,

2009). This variation can stem from sampling error (i.e., within-study variability), which is always present in a meta-analysis, because every single study uses different samples

(Huedo-Medina, Sánchez-Meca, Marín-Martínez, & Botella, 2006), or systematic sampling error. Additionally, between-studies variability, which can appear when the true heterogeneity exists among population effect sizes estimated by each study due to the influence characteristics that vary among the studies (e.g., characteristics of samples, variations in treatments, variations in the design quality, etc.)(Huedo-Medina et al.,

2006). Therefore, it becomes necessary to determine what part, if any, of the observed variation is real (Borenstein et al., 2009). Several measures are possible for estimating the level of heterogeneity. One of the most common within meta-analysis I2 index, initially proposed by Higgins and Thompson (2002), can be calculated to investigate potential heterogeneity.

Step 6: Interpreting the Evidence

Cooper (2017) recommends several steps in interpreting the evidence gathered for analysis in a meta-analysis. Exploring “missing data” coincides with taking aims to avoid the “file drawer problem” (Rosenthal, 1979), which results from non-publishing of statistically non-significant results. Statistical sensitivity analysis allows for the comparison between the different models of meta-analysis (Cooper, 2017). Finally, meta-

108

analysts may examine and apply descriptors to the size and practical significance of the calculated effect size. Each of these steps are important for this type of research.

Step 7: Presenting the Results

Finally, valid and complete reporting of the results of any meta-analysis is crucial to its understanding and application. Cooper (2017) explains that variations in reporting can lead to questioning the results of the meta-analysis and/or influence the replication of the process. Tamim et al. (2011) specifically expound the importance of methodological quality in meta-analysis work, stating:

Unlike the report of a single primary study, a meta-analysis carries the weight of a whole literature of primary studies. Done well, its positive contribution to the growth of a field can be substantial; done poorly and inexpertly, it can actually do damage to the course of research (p. 12).

There are multiple criterion references available for this process. To evaluate meta- analytic study quality, Cooper (2017) recommends the meta-analytic reporting standards

(MARS) created by the American Psychological Association (APA), which examine the following criteria:

 Title  Abstract  Introduction  Method o Inclusion and exclusion criteria o Moderator and mediator analyses o Search strategies o Coding procedures o Statistical methods o Results o Discussion

A full description of the APA MARS is provided in Appendix B.

Alternately, Moher, Liberati, Tetzlaff, Altman, and Group (2009) generated the

PRISMA Checklist, a document providing examples of good reporting, a rationale for each components inclusion, and supporting evidence, including references, whenever 109

possible (p. 3). The checklist (provided in full in Appendix C) recommends consideration of the following:

 Title  Abstract  Introduction o Rationale o Objectives  Methods o Protocol and registration o Eligibility Criteria o Information sources o Search o Study selection o Data collection process o Data items o Risk of bias in individual studies o Summary measures o Synthesis of results o Risk of bias across studies o Additional analyses  Results o Study selection o Study characteristics o Risk of bias within studies o Results of individual studies o Synthesis of results o Risk of bias across studies o Additional analysis  Discussion o Summary of evidence o Limitations o Conclusions  Funding

Both the PRISMA Checklist and MARS criterion are appropriate guidelines to insure the quality of meta-analytic research.

Study Methods

To better understand the relationship of teacher education courses and second- order barriers related to technology integration, a series of meta-analyses were conducted, along with corresponding secondary analyses. Detailed in the following sections are the

110

methods specific to the research within this dissertation. Framed through Cooper’s (2017) seven steps, the methods below were managed.

Step 1: Formulating the Problem

To formulate this meta-analysis’ focus, the research here answered Cooper (2017) three questions:

1. Should the results of the research be expressed in numbers or narrative?

2. Is the problem you are studying a description of an event, an association between events, or a causal explanation of an event?

3. Does the problem or hypothesis seek to understand (a) how a process unfolds within an individual participant over time, or (b) what is associated with or explains variation between participants or groups of participants (p. 42)?

This research is effectively explored through both numbers and narrative. The exploration of the relationship between TECTI and barriers to technology integration was explored using a systematic review of the corresponding literature and a statistical evaluations of the relationships of effect sizes across studies. Addressing the second question, this study sought to describe the relationship between TECTI and the second-order barriers. No associative or causal conclusions were made on results of this study. Finally, the research sought to understand how the process unfolded in the participant enrolled in the TECTI.

A description of this process was the simplified goal of this research.

Step 2: Searching the Literature

The search plan for this research used five major search engines (i.e., EBSCO,

ERIC (ProQuest), LearnTechLib, Science Direct, and ProQuest Dissertations & Theses

Global) to identify potential articles. Specific collections databases for education research included within the EBSCO search were Academic Search Premier and Education

Source; and these specific databases were used to refine the search. Both EBSCO and

ERIC(ProQuest) were chosen for their deep repositories of educational research.

111

LearnTechLib was selected because of its high percentage of educational technology articles, including conference proceedings. Additionally, Science Direct was chosen due to a high return rate of educational technology articles. Boolean search strings and potential filters were formulated per the search service used (see Table 3-2 below). These initial searches yielded 1285 studies from EBSCO (302 and 983 from Academic Search

Premier and Education Source, respectively), 556 from ERIC, 551 from LearnTechLib,

184 from Science Direct, and 360 from ProQuest Dissertations & Theses Global. The initial search resulted in 2,936 total documents for initial consideration, which were consolidated using the RefWorks Legacy platform (hereafter RefWorks). Of the 2,936 initial studies, 121 exact duplicate references as identified by the RefWorks.

Table 3-2. Search criteria. Database Boolean String Filters EBSCO TX (((technology OR ICT) AND (integration OR Limiter - Published integrated OR integrate OR integrating)) AND Date: 20070101- (“preservice teacher” OR “teacher education” OR 20171231 “teacher preparation”) ) AND AB ( attitude OR Expander - Also search confidence OR efficacy OR stress OR belief OR within the full text of the knowledge OR TPACK OR TPCK ) AND AB ( articles “preservice teacher” OR “teacher education” OR Search mode - “teacher preparation” ) Boolean/Phrase ERIC(ProQuest) (((ICT OR technology) AND integrat*) AND NA ("preservice teacher" OR "teacher education" OR "teacher preparation")) AND ab(attitude OR belief OR knowledge OR TPACK OR TPCK) AND ab("preservice teacher" OR "teacher education" OR "teacher preparation") LearnTechLib (((technology OR ICT) NEAR/5 (integration OR Date (2007 - 2018) integrating OR integrated)) AND ("preservice Full text teacher" OR "teacher education" OR "teacher preparation")) Science Direct ((technology AND ICT) AND (integrate OR pub-date > 2006 integration OR integrating)) AND (“preservice teacher” OR “teacher education” OR “teacher preparation”) ProQuest Dissertations ft((ICT OR technology OR "technology NA & Theses Global integration" OR "integration of technology") AND ("preservice teacher" OR "teacher education" OR "teacher preparation")) AND ab(attitude OR belief OR knowledge OR TPACK OR TPCK) AND ti(attitude OR belief OR knowledge OR TPACK OR TPCK) AND ab("preservice teacher" OR "teacher education" OR "teacher preparation")

112

After studies were identified, information a systematic process of study information gathering carried out through multiple stages. This process is explained in the following section.

Publication bias is a known issue within meta-analyses. To estimate the potential of publication within the meta-analyses in this study, both a trim-and-fill funnel was plotted, as recommended by Duval and Tweedie (2000a, 2000b), and three Fail-safe N statistics calculated. Each of these were conducted following the information gathering process detailed in the following section. Those results are presented in Chapter 4, and implications discussed in Chapter 5.

Step 3: Gathering Information from Studies

Inclusion/exclusion criteria tie conceptual variables to observable research operations and measurements (Cooper, 2017). Identified studies qualified for this meta- analysis, if:

 The study examines the impact of the course on at least one of the three identified second-order barriers to technology integration (i.e., attitudes, beliefs, and/or knowledge).

 The study had an empirical comparison of change in a related outcome variable as defined in this meta-analysis.

 Participants are preservice teachers participating in a four- or five-year, university-based teacher education program.

 Participants are preservice teachers participating in a stand-alone course on technology integration.

 The study reported data or indications that allowed calculation or estimation of effect sizes for study effects (e.g. means and standard deviations, t-score, F-score etc.).

 Studies were published or presented starting in 2007 and beyond.

 Articles were freely available through the University of Florida library system and its affiliates.

113

 The article is in English.

To identify potentially qualifying studies, the process was as follows.

The first stage involved coding abstracts and titles of the identified works. To begin the coding process, the references of identified studies were exported from

RefWorks to a comma-separated values file (CSV) for manipulation in Excel. Using and

Excel function, alphanumeric codes (two letters and two numbers) were randomly generated. Unique codes were verified through duplication identification tools available in Excel. Each of the references was assigned one unique code. Post code assignment, the file was sorted alphabetically by code to randomize the reference list. A co-coder was recruited among the doctoral students in the educational technology program at the

University of Florida. This student was selected for her expertise in meta-analysis to aid in the process of coding and to verify the reliability of the coding process. All future references to the co-coder refer to this person. Abstract and title coding took place in

Qualtrics (see tool in Appendix D). Both the primary researcher and the co-coder coded at this stage. Each coder maintained a separate Qualtrics tool for coding to allow for coder identification. The co-coder was assigned to code the first 10% of the references (n

= 282). The first thirty (n = 30) of those by code alphabetically were used for coding training. A Cohen’s kappa (Cohen, 1960) was calculated to quantify the inter-rater reliability (IRR). IRR of this portion of the coding process was 0.568, which can be interpreted as a moderate agreement in raters (McHugh, 2012). The primary researcher and the co-coder discussed the coding process, and the moderate agreement was most likely attributed to a difference of understanding on how the outcome variables may be presented in studies. All remaining title and abstract coding was completed by the primary researcher.

114

The title and abstract coding process conducted yielded 465 references that potentially contained data relevant to the main study. Studies recommended for inclusion by the co-coder, but initially rejected by the primary researcher, were reviewed. An additional four studies (n = 4) recommended for inclusion by the co-coder were added back into the study pool for an aggregate total of 469 studies. This number was reduced to 383 by removing references that fell outside the date range for the current study (i.e.,

2007 and beyond). Subsequently, deduplication was re-conducted using duplication matching in Excel post abstract and title coding. Through the coding process, the primary researcher noticed the duplication of title and abstracts, and recognized that the

RefWorks deduplication process was not foolproof. Using Excel, duplicate titles or author names were highlighted, and visually scanned to identify duplicate titles. An additional visual scan was also conducted as some author names or titles were modified by RefWorks to maintain database entry uniqueness. The result was 326 remaining references retained for the next round of coding.

The second round of coding explored more specifically the available content of the articles. This round was set to verify that the reference looked at attitudes, beliefs, and/or knowledge connected to technology integration; that the study took place during a stand-alone technology integration course for preservice teachers; and the article included quantitative data from which effect sizes may be calculated (e.g., means and standard deviations, t-scores, etc.). Additional, the identification of unpublished studies, also known as grey literature, was addressed at this stage. Items in this stage being retained for full coding were searched for additional studies potentially falling within the scope of this study. Reference lists were searched to implement a backward search identification.

Also, a forward search was conducted using Web of ScienceTM. Titles or authors were

115

searched via the Web of ScienceTM citation search function. The references resulting from Web of ScienceTM were either searched in full, or, if the work yielded more than 25 forward references, the first 25 citations by usage count since 2013 were examined. Both decisions here were made to create a manageable work load, while focusing on the most actively cited research. Alternately, some references retained after abstract/title coding stage were either book chapter or conference proceeding lists. These were coded under a separate protocol to verify if any of the list contained potential titles for coding. The coding protocol for this stage can be seen in Appendix E.

The 326 articles were divide into approximately thirds for coding. The first third were coded by both coders for training and inter-rater reliability (n = 108). The IRR for this stage yielded a Cohen’s kappa of .618, a substantial agreement (McHugh, 2012).

Post-coding a discussion took place between the coders to work out discrepancies in the coding process. The most common disagreements came in the areas of course description or outcome variable description. The coders came to full agreement after this discussion.

Subsequently, a third each were coded by the individual coders to complete the process.

Seventy-seven (n = 77) articles were retained as being relevant to the study. Additionally,

151 new, unique articles resulted from the forward and backward citation search, which required additionally coding. These articles were strictly coded through the second-round process by the primary coder, excluding further forward/backward citation searching.

Forty-three (n = 43) articles were retained for full coding from that round of coding. The final count of articles retained from the second stage of coding was 120.

The final stage of coding was conducted using the full coding protocol. As per

Cooper (2017) recommendations, a pilot test of the codebook with the co-coder was conducted to revise for greater clarity in questions, responses, and conventions of the

116

coding process. A draft version of the protocol (Appendix F) was developed based on the framework of this study. Additionally, 10% of the potential studies identified in the prior stage were examined for features included in their courses to identify as many potential course features as possible. These features were added to the draft version of the final coding protocol. Once created, this version of the protocol was discussed with the co- coder to identify any unnecessary items do to redundancy or lack of need, and for training purposes. Some items (e.g., authors, publication date, etc.) were deemed unnecessary to code for because the information was present in the data files from RefWorks, while others (e.g., coder ID, time on coding, etc.) would either collected by the Qualtrics system or managed through separate coding tools for each coder. The tool was developed in Qualtrics. The finalized version of the protocol presented at the end of this document

(Appendix G), as well as the codebook (Appendix H). Overall, this stage of coding examined in full the availability of effect size data, study quality, the presence or absence of sub-group variables, the validity of tools used in the study, and the reliability of the data within the study. Each coder undertook the coding of 25% (n = 30) items to check inter-rater reliability. The result of this check yielded a kappa value of 0.619, or a substantial agreement (McHugh, 2012). A discussion was held on the items under disagreement, and a consensus was reached. Additionally, at this stage it was determined that any study that used a design focused on correlational outcomes would not be appropriate for this research. Therefore, any correlational study was determined to be inappropriate for this research, and coded as such moving forward. All further coding was conducted by the primary researcher due to scheduling conflicts for the co-coder.

The final coding stage resulted in the inclusion of fifty-five (n = 54) total articles across

117

all outcome variables. Attitudes, beliefs, and knowledge were examined in 28, 12, and 38 articles each, respectively.

After the full coding stage, some of the features in the coding process needed to be recoded. Initially, reliability was examined with the lens that it was purely established in the measurement. This did not provide the needed distinction between studies. The decision was made to explore the distinction between reliability in studies where it was statistically established. Additionally, multiple measures were used in some reports, and so this could not be answered at the study level, but needed to be addressed at the variable level. Similarly, validity was also difficult to assess at the study level for the same reason. Therefore, both these features were recoded through Excel once the data was extracted from Qualtrics.

After finalizing the identification of the studies for inclusion, effect size (ES) data was extracted from each study. When available, means and standard deviations (SDs) were extracted. In other cases, the primary researcher extracted t-scores, F-scores, or correlations for ES calculation. For those studies with means and SDs, ESs were calculated in R (R Core Team, 2017) using the metafor package (Viechtbauer, 2010). The metafor package calculated ESs and applied the appropriate adjustments for Hedges’ g.

In the case of studies reporting other data, ESs were calculated using the tools at campbellcollaboration.org’s Practical Meta-Analysis Effect Size Calculator developed by

Dr. David B. Wilson at George Mason University from the formulas in book on meta- analysis by Lipsey and Wilson (2000). To apply the Hedges’ g adjustment for these studies, the J was calculated and applied in an Excel spreadsheet.

There are multiple issues that can cause problems with the gathered data. Cooper

(2017) identifies incomplete reporting of statistical outcomes and incomplete reporting of

118

other study characteristics as primary issues related to meta-analysis related to study retrieval. To address shortcomings in potentially fruitful research, one of four methodological steps was applied in the present meta-analysis. First, if statistical outcomes are underreported (e.g., missing means and standard deviations, missing sample sizes, etc.) an attempt was made to contact the study authors to request the information.

Second, if the information request was ignored, a search was conducted for a related article on the same data study. Finally, if the aforementioned methods failed to yield the needed data, an assessment was made to determine if it was appropriate to assume a null result. In the single case of Clark, Zhang, and Strudler (2015), a null result was imputed for the effect size of the attitude variable, because the authors simply reported that there was no statistical significance to that variable. An estimate of the variance for this effect size was calculated from the other measures in the study. Finally, if none of the aforementioned options were viable, then the study was excluded. No studies were excluded for this reason.

Several of the studies identified for final inclusion in the current research included multiple measures used to calculate effect size or examined multiple populations within the same study. These conditions were addressed in the following two ways. First for those studies providing multiple effect sizes from the same population, Borenstein et al.

(2009) recommend aggregating the effects by finding the mean ES and calculating the variance of the combined effects using a robust variance estimation (RVE). The reasoning here is that multiple measures of the same problem creates a violation of the assumption of independence. To that end, each study using multiple measures had the

ESs aggregated through the method prescribed for RVE by Borenstein et al. (2009) in their book using the MAd package (Del Re & Hoyt, 2014) in R. The correlational value

119

for the aggregation of these ESs was set to 0.75, because each of the measures of the outcome variable, while not identical, were probably at least moderately correlated across studies. For those studies with multiple study populations, each independent population was treated as a separate study, and the ESs, variance, etc. were calculated for each identified study. This yielded a study count in the current research of 34 studies on attitudes, 12 studies on beliefs, and 46 studies knowledge examined.

Step 4: Evaluating the Quality of Studies

Evaluating study quality can involve a priori or a posteriori of meta-analysis design. A priori quality was addressed through inclusion/exclusion criteria. To assess study quality a posteriori, potential quality control issues were identified via Conn’s and

Rantz’s (2003) criterion suggested for managing the primary study quality for meta- analyses, and appropriately adapted for this study (see Table 3-3).

Table 3-3. Preliminary criterion for study quality. Adapted from Conn and Rantz (2003) Concept Issues Sample selection Intervention tested with important sub-groups. Recruitment Recruitment strategy prevents bias. Description of potential subjects who declined participation. Sample size adequacy Size adequate to provide a sufficiently precise estimate of effect size. Random assignment Central system generates an unpredictable assignment sequence. Allocation concealment/randomization blinding. Comparison grouping Nature of comparison group appropriate for the area of science. Management of pre-intervention differences between comparison groups. Blinding/masking Participants Instructor Assessor measuring outcome Data analyst Interventions Intervention reproducible by others Intervention consistent with theory Treatment integrity Attrition management Attrition prevented and reported Outcome measures Construct validity of instrument ascertainable Adequate reliability to provide sufficiently precise estimate of effect size Appropriate follow-up period to measure outcomes Statistical analysis Assumptions of analysis consistent with data Significance level appropriate given number of tests conducted on data Potential confounders not controlled in design addressed in analysis Exact test statistic values and p levels presented

120

Additionally, Gersten and Edyburn (2007) proposed a rubric, specifically for special education technology research, evaluating eight quality control areas (e.g., sample selection, intervention implementation, outcome measures, etc.) in research reporting.

Through a synthesis of these two resources, a checklist was generated to evaluate the quality of the primary studies considered for this meta-analysis, which was integrated into the coding protocol. Quality features were added to the coding guide for evaluation and rater comparisons.

Step 5: Analyzing and Integrating the Outcomes of Studies

Effect size estimation and aggregation

In the case of studies reporting other data, ESs were calculated using the tools at campbellcollaboration.org’s Practical Meta-Analysis Effect Size Calculator developed by

There are multiple issues that can cause problems with the gathered data. Cooper

121

methodological steps was applied in the present meta-analysis. First, if statistical outcomes are underreported (e.g., missing means and standard deviations, missing sample sizes, etc.) an attempt was made to contact the study authors to request the information.

Second, if the information request was ignored, a search was conducted for a related article on the same data study. Finally, if the aforementioned methods failed to yield the needed data, an assessment was made to determine if it was appropriate to assume a null result. In the single case of Clark et al. (2015), a null result was imputed for the effect size of the attitude variable, because the authors simply reported that there was no statistical significance to that variable. An estimate of the variance for this effect size was calculated from the other measures in the study.

ESs aggregated through the method prescribed for RVE by Borenstein et al. (2009) in their book using the MAd package (Del Re & Hoyt, 2014) in R. The correlational value for the aggregation of these ESs was set to 0.75, because each of the measures of the outcome variable, while not identical, were probably at least moderately correlated across studies. For those studies with multiple study populations, each independent population was treated as a separate study, and the ESs, variance, etc. were calculated for each

122

identified study. This yielded a study count in the current research of 34 studies on attitudes, 12 studies on beliefs, and 46 studies knowledge examined.

Step 6: Interpreting the Evidence

Main effect estimation

The mean effect size of each attitude, beliefs, and knowledge within this study were calculated using a random-effects model to all outcome variables. Additionally, each study was weighted by the inverse of its variance. Borenstein et al. (2009) explains the weight of each study is:

1 푊∗ = (3-13) 푖 푉∗ 푌푖 where ∗ is the within-study variance for study (i) plus the between-study variance, T2 푉푌푖

(p. 73). This formula was used to calculate all relevant weights. From there, the weighted mean, M*, was computed using the formula:

∑푘 ∗ ∗ 푖=1 푊푖 푌푖 푀 = 푘 ∗ (3-14) ∑푖=1 푊푖

(Borenstein et al., 2009, p. 73). Further, Borenstein et al. (2009) explain that the variance of the summary effect is estimated with the formula:

1 ∗ 푉푀 = 푘 ∗ (3-15) ∑푖=1 푊푖 and the estimated standard error of the summary effect is the square root of the variance

푆퐸푀∗ = √푉푀∗ (3-16)

Finally, the 95% confidence interval can be calculated using:

퐶. 퐼. = 푀∗ ± (1.96 (3-17) × 푆퐸푀∗)

123

(Borenstein et al., 2009, p. 74). To calculate the standard error of the summary effect, the following formula was applied:

휎2 휏2 푆퐸 = √ + (3-18) 푀 푘 × 푛 푘

(Borenstein et al., 2009).

Both a fixed-effect model (FEM) and random-effects model (REM) were calculated for each outcome using the data. However, an a prior choice to use the REM was made because of the high probability of variability among the studies. For example, potential variable methods included differences in populations, measurement tools, and course structures and content. This was supported by the outcomes of the analyses. Using the metafor package in R, the REM model with calculated using the restricted maximum likelihood (REML) method. All results are reported in the Chapter 4.

Sub-group analyses

The sub-group analyses within this study applied a random-effects model to all outcome variables. The sub-group analyses looked at the study conditions including course features, study quality, reliability within the study, and validity of the measurement tool. Course features identified prior to and during the coding process yielded eight potential features: mentoring/coaching, rehearsal/field experience, goal- setting, observation, reflection/self-evaluation, hands-on learning, work sample analysis, and practice lesson planning. Each study was coded for the presence (1) or absence (0) of the course feature.

124

Study quality was a composite score of multiple items within the coding process.

The quality control items, as well as publication and peer-review, were summed to get a score. From there, a quartile ranking of the quality scores was calculated for each outcome variable’s scores. Those studies falling in the lowest quartile were coded as being of “low” (0) quality. Conversely, all studies in the highest quartile were coded as

“high” (2). All others were coded as having a standard or average quality score (1). These quality scores rankings were used for sub-group analyses.

Reliability of a study was coded as a function of the individual study. Many studies reported reliability scores from prior studies. As such, for this meta-analysis it was determined that a better reliability identifier was if the study checked the reliability of study data. Each study was coded for if the study reported reliability data for their research (1), or failed to report such data (0).

Finally, validity of the studies was examined based on the tools selected for measurement. There were three potential conditions. The first condition, coded “0,” was assigned for those studies that used a study specific tool or tools with no prior validation.

The second condition, coded “1,” was assigned for those studies that used tool or tools for measurement that had been previously validated, but then were modified in some manner by the researcher (e.g., translation, only selecting a portion of the items, etc.).

The final condition, coded “2,” was assigned to studies where a previously validated tool or tools was used without modification.

All sub-group analyses used a REM. As with the main effects, an a prior choice to use the REM was made because of the high probability of variability among the studies. This means that the analyses were conducted by calculating separate estimates of tau for each group, as per the flowchart (Figure 3-2) earlier in the chapter (Borenstein et

125

al., 2009). As with the main effects, the metafor package in R was used to compute the models within the sub-groups, and restricted maximum likelihood (REML) method was used for the estimation. While each sub-group had balance of cases, some moderator variables yielded only a single or no alternate case for analysis. The outcome variable belief yielded no studies with goal-setting as a course feature, and as such made sub- group analysis moot. Meanwhile, course features with an alternate case identified only in a single study (e.g., there was only a single case of mentorship/coaching identified in the belief studies) caused the metafor package in R to default to a FEM for computations. All results are reported in the Chapter 4.

Heterogeneity analysis

As previously stated, quantifying the homogeneity, or its opposite, heterogeneity, is vital to understanding the results of meta-analyses, because the dispersion in observed effects is partly spurious (Borenstein et al., 2009). This variation can stem from sampling error (Huedo-Medina et al., 2006), or systematic sampling error. Additionally, between- studies variability, which can appear when the true heterogeneity exists among population effect sizes estimated by each study due to the influence characteristics that vary among the studies (e.g., characteristics of samples, variations in treatments, variations in the design quality, etc.) (Huedo-Medina et al., 2006). This is to be expected within the present meta-analyses because of the differences in populations, course design, and study outcomes for example. To determine what part, if any, of the observed variation is real (Borenstein et al., 2009), the I2 index, initially proposed by Higgins and

Thompson (2002), was calculated to investigate potential heterogeneity within the meta- analyses conducted here. This was calculated as part of the analyses run through the metafor package.

126

Step 7: Presenting the Results

The results of this series of meta-analyses are reported within this dissertation report. Both the APA MARS Checklist and the PRISMA Checklist for meta-analysis quality were consulted throughout the process to assure all needed components of reporting were included in this dissertation. The process of the systematic review of literature and meta-analyses, both main and sub-group are presented above. Meta- analysis results and discussion of those results are included in the following two chapters.

Conclusion

Employing Cooper’s (2017) seven step process for systematic review and meta- analysis, an investigation into TECTI impact on second-order barriers was conducted through a series of meta-analyses. Research into the topic was systematically identified, coded, and mined for relevant statistical information. Main effects of TECTI effect on attitudes, beliefs, and knowledge were calculated. Sub-group analyses into the role that course features, reliability within a study, study quality, measurement validity, and reported reliability were also conducted. Each of these analyses are presented in the next chapter.

127

CHAPTER 4 RESULTS

The process of the systematic review of the literature and meta-analysis were used to investigate several questions as to the impact of TECTI on the attitudes, beliefs, and knowledge of preservice teachers for technology integration. Specifically, the research examined the effect of TECTI on second-order barrier constructs (i.e., teacher attitudes, beliefs, and knowledge). Additionally, subgroup analyses explored:

 What is the difference in magnitude of effect of TECTI on each second-order barrier construct when comparing levels of overall study quality?

 What is the difference in magnitude of effect of TECTI on each second-order barrier when comparing levels of measure validity?

 What is the difference in magnitude of effect of TECTI on each second-order barrier when comparing levels of reported reliability?

What follows is a description of the systematic review process, an examination of potential publication bias, and the results of the main effect meta-analyses and all sub- group analyses, including heterogeneity statistics.

Systematic Review

The systematic review process and its results have been described in full in

Chapter 3 of this dissertation. Therefore, a summary of the process and the results are presented here. The review process began with collecting articles across six databases, which yielded 2,936 studies. Through three stages of coding (i.e., title/abstract coding, primary study feature coding, and full coding) the viable studies were narrowed to a total of 55 studies across all three second-order barrier variables. Inter-rater reliability values

(Cohen’s kappa) at each of the stages was 0.568, 0.618, and 0.619, which suggest that at

128

each stage the agreement between the two coders was acceptable. Because of multiple populations examined within several studies, the final study count for each attitudes, beliefs, and knowledge were 27, 12, and 38 studies, respectively. A PRIMA flowchart of this process is provided in Appendix I.

Description of the Studies

The following sections track the nature of the studies under review for this study.

Study Publication Description

The studies for the current research were drawn from a variety of resources all published between 2007 and 2017. While the total number of publications were fifty-four

(n = 54), several of the publications yielded multiple outcome variables and/or multiple populations under examination. As such, the counts represented here are the number of study populations under examination for each variable. Tables 4-1 through 4-3 detail the types of publication for each study variable.

Table 4-1. Attitudes studies. Publication Type Count Percentage Journal 24 70.6% Conference proceeding 5 14.7% Dissertation 4 11.8% Book Chapter 1 2.9% Totals 34 100%

Table 4-2. Beliefs studies. Publication Type Count Percentage Journal 9 75% Conference proceeding 2 16.7% Dissertation 1 8.3% Totals 12 100%

Table 4-3. Knowledge studies. Publication Type Count Percentage Journal 33 71.7% Conference proceeding 5 10.9% Dissertation 7 15.2% Book chapter 1 2.2% Totals 12 100%

129

As can been seen, a majority of the studies for each variable came from peer- reviewed journals. This means that the research within had been through a rigorous screening process. Beyond that, dissertation reports and conference proceedings have an approximately equivalent examination of each variable. Book chapters lent only single studies to the attitudes and knowledge variable.

Study Population and Context Descriptions

The context of a study and a description of the population provide rich context for the interpretation of this research. Presented here are three components of the context and population. The number of credit hours allows for an understanding of the nature of exposure to course related concepts, which may influence the degree of impact the course had on one or more of the variables. Next, the program level, either undergraduate or graduate, helps to understand the maturity of though related to concepts and practical application of educational pedagogies and philosophies. Finally, the location of the study better allows for the interpretation of the nature of technology integration. This is not to say conclusively that there is a difference in conceptualization and practical application of technology integration for teaching and learning in these contexts. However, there is a potential for this that must be acknowledged. Tables 4-4 through 4-6 detail the results of each of these contextual and population features.

Table 4-4. Course credit hours of technology integration courses in primary sources. Number of Credit Hours Study Counts and Percentages Attitudes Beliefs Knowledge Unknown 14 (41.2%) 8 (75%) 25 (54.3%) 1 6 (17.6%) NA 4 (8.7%) 2 10 (29.4%) 2 (16.7%) 11 (23.9%) 3 3 (8.8%) 1 (8.3%) 5 (10.9%) 4 NA 1 (8.3%) NA 5 NA NA NA More than 5 1 (2.9%) NA 1 (2.2%) Totals 34 (100%) 12 (100%) 46 (100%)

130

Table 4-5. Course level of technology integration courses in primary sources. Course Level Study Counts and Percentages Attitudes Beliefs Knowledge Undergraduate 33 (97.1%) 10 (83.3%) 42 (91.3%) Graduate 1 (2.9%) 2 (16.7%) 4 (8.7%) Totals 34 (100%) 12 (100%) 46 (100%)

Table 4-6. Study location of technology integration course university in primary sources. Study Location Study Counts and Percentages Attitudes Beliefs Knowledge N. America 26 (67.5%) 6 (50%) 28 (60.9%) Africa 1 (2.9%) NA 3 (6.5%) Europe 4 (11.8%) 3 (25%) 5 (10.9%) Asia 3 (8.8%) 3 (25%) 10 (21.7%) Totals 34 (100%) 12 (100%) 46 (100%)

Understanding nature of the context and study population aids in the interpretation of the results of the meta-analyses within this research. As can be seen, there are some important features to be considered when reading and applying the results here. First, at best it could be said there is a vagueness to the nature of the courses due to the lack of reporting in the research. For each variable, a majority percentage of the study reports lack a clear definition of the credit hours of contact between the students and the concepts related to technology integration. Meanwhile, for those studies that do report the number of credit hours, the hours range from as few as 1-credit hour to over 5-credit hours for some variables. Looking at the mode and mean for each of the variables, it appears as if the typical number of contact hours is about 2-credit hours for technology integration courses in these studies.

Next, the program level, either undergraduate or graduate, helps to understand the maturity of though related to concepts and practical application of educational pedagogies and philosophies. Each of the primary variables for this research was typically explored in undergraduate courses with on average less than 10% of the studies examining graduate level populations. The focus on technology integration primarily at the undergraduate level may potentially mean that the students lack the maturity to

131

interpret their ability to integrate technology, which could influence features of the following section of this report looking at the methodologies and measurement tools in the primary studies.

Finally, while it cannot be stated conclusively that there is a difference in conceptualization and practical application of technology integration for teaching and learning in different study contexts, there exists a potential for such differences that must be acknowledged. A majority of the primary studies were conducted in North American

(i.e., American and Canadian) universities. The second and third most likely locations for these studies were either Europe or Asia (from the Middle East to East Asia) depending on the primary variable. Finally, African universities contributed study populations to a few studies for the attitudes and knowledge variables. Do these different contexts create differences in theoretical frameworks? There is no way to say conclusively from this study, but there may potentially exist a difference.

Methodological and Measurement Tool Descriptions

Like population and contextual features, understanding the design of a study helps better interpret the results of any research. Presented below are two features of the primary studies under consideration in this research: the study design of the primary studies and the self-report nature of the measurement tools. Knowing the design of the study aids in interpreting the comparisons between different primary studies in the meta- analyses. Looking at the nature of the measure for self-reporting on the variable helps to decipher to a degree the change in variable by study’s end. Tables 4-7 and 4-8 look at study design and self-report measurement tool, respectively.

132

Table 4-7. Study design of the primary source studies. Study Design Study Counts and Percentages Attitude Beliefs Knowledge Pre-Post Test 29 (85.3%) 9 (75.0%) 41 (89.1%) Experimental Pre-Post Test 3 (8.8%) 2 (16.7%) 2 (4.3%) Experimental 2 (5.9%) 1 (8.3%) 2 (4.3%) Quasi-experimental design NA NA 1 (2.2%) Totals 34 (100%) 12 (100%) 46 (100%)

Table 4-8. Self-reported measures in primary source studies. Self-Report Measure Used Study Counts and Percentages Attitudes Beliefs Knowledge Yes 34 (100%) 12 (100%) 37 (80.4%) No 0 (0%) 0 (0%) 9 (19.6%) Totals 34 (100%) 12 (100%) 46 (100%)

To better understand the comparisons between studies and changes occurring in the variable within primary studies, those features were reported here. While in each variable was primarily explored using a pre-/post-test design, a percentage of the studies in each case utilized other methods. This could potentially amplify the inherent “apples to oranges” comparison problem with meta-analyses in this present study. Alternately, the high percentage of pre-/post-test design requires an acknowledgement of the shortcomings of said design. While this design is known to improve internal validity, it typically comes at a sacrifice to external validity (Salkind, 2010). This means that the results are not comparable to groups that are untreated, and as such the result should be interpreted in light of this.

The instrumentation of the studies under consideration can also illuminate the nature of the research. Beginning with attitudes, the tools employed in the studies typically measured self-efficacy, stages of adoption, general computer attitude, and anxiety. Ten of the 30 studies not using study created measures used the Computer

Technology Integration Survey (CTIS) instrument from Wang et al. (2004), thereby making it the most prevalent tool in this area. The second and third most commonly used tools were the Attitudes Toward Computer Technology (ACT) instrument (Kinzie et al.,

133

1994; Milbrath & Kinzie, 2000) and the Teachers’ Attitudes toward Computers questionnaire (Christensen & Knezek, 2000). Other tools were only use in one or two studies each, and were typically newer or region specific.

For beliefs, there were no consistently used measures. Each measure was used in a single case. This may account for some of the widespread heterogeneity of this variable.

Finally, knowledge was looked at in terms of ICT skills or TPACK knowledge.

The Schmidt et al. tool from 2009 was the tool of choice in research identified for this study. Thirteen of the 35 studies used for this research utilized the Schmidt et al. tool.

The next most commonly used tool was a TPACK survey developed Koehler and Mishra

(2005). Other tools used throughout the studies were either based on TPACK or came from state agencies or national associations (e.g., ISTE).

Another important consideration is the nature of these measurement tools. As presented above, two of the three variables considered in this research fully utilize self- reported measures, while in the third over 80% of the studies also used such measures.

There exist several inherent flaws to self-report measure which must be considered when interpreting the results of the meta-analyses within the current research. Hoskins (2012) highlights self-reporting lends itself to issues of reporting honesty and introspection by the respondent. Participants may rate themselves higher than they are or report change because they wish to appear better than they are or because they lack the ability to self- evaluate on the topic. Also, because of the design of self-report measures, typically employing some form of rating scale, there may be issues with how people respond and/or the vagueness of the scale differences (Hoskins, 2012). People inherently judge the meaning of scales differently, and therefore the self-ranking along the scale may not be universally constant. Furthermore, rating scales utilize ordinal rankings, which while

134

defined as to meaning, really create no definitive differences between the ranking. In other words, is the difference between a “strongly disagree” to “agree” the same as the difference between “agree” to “strongly agree?” This imprecision makes the interpretation of studies using such measures difficult in many cases.

Publication Bias

The potential for publication bias was investigated through both a trim-and-fill plot (Duval & Tweedie, 2000a, 2000b), and the calculation of a Fail-safe N (Dickersin,

Rothstein, Sutton, & Borenstein, 2005). Potential publication bias in this meta-analysis was treated separately for each of the three second-order barrier variables (i.e., attitudes, beliefs, and knowledge). Presentation of the plots is presented first with the Fail-safe N calculations to follow.

Trim-and-Fill Plots

Trim-and-fill plots, suggested by Duval and Tweedie (2000a, 2000b), through an iterative process create plots of potentially missing articles that could significantly shift the outcome of the meta-analysis (Duval, 2005). The black plot points represent existing studies in the pool of identified research. The white plot points represent potentially missing studies, which, when present, would balance the studies around the mean thereby lessening the potential bias. The potential publication bias of the three variables’ studies are presented in turn. First, Figure 4-1 shows the for the attitude variable.

135

Figure 4-1. Attitude variable trim-and-fill

As can be seen, the plot indicates for the studies identified for attitude that the study is asymmetrical, and appears to be lacking in higher end values. The overall dispersion may be attributed to a number of factors, but the most likely in this portion of the study in the standard errors impact on the confidence intervals of the study. The plot suggests there is likely some bias in this portion.

The second plot presented looks at the potential bias in the belief outcome (Figure

4-2). As with attitude, the belief outcome is potentially lacking in studies above the mean, as well. Additionally, the low number of studies and the overall high standard error suggests that bias may be impacting the outcome of this meta-analysis.

136

Figure 4-2. Belief variable trim-and-fill.

Finally, Figure 4-3 presents the trim-and-fill estimation for the knowledge outcome. Unlike the prior two outcomes, knowledge may be lacking in studies beneath the mean difference. This is logical due to the high number of large, positive ESs connected to this outcome. However, the large grouping of studies about the mean with low standard error, and the fairly symmetrical distribution of the studies indicates a low potential bias impacting this analysis.

137

Figure 4-3. Knowledge variable trim-and-fill.

From the trim-and-fill plots, the likelihood of bias appears be low for knowledge.

Conversely, the belief and attitude results may be biased. This is based purely on the visual examination and interpretation of the proceeding plots. The subjective nature of the interpretation of the plots is clarified through the quantitative methods in the following section.

Fail-Safe N Results

Due to the indication of the trim-and-fill plots that publication bias may exist within this research, calculation and interpretation of a Fail-safe N importantly quantifies a threshold for when such bias becomes problematic. As stated last chapter, Rosenthal’s method (1979) is used to compute how many missing studies would need to be retrieved and incorporated in the analysis before the p-value became nonsignificant (Borenstein et al., 2009). Orwin’s (1983) proposed method allows control of the target effect size. For the calculations here, the “target” effect size of substantive effect was estimated by the metafor package. The target effect sizes were 0.321, 0.141, and 0.100 for each attitude, 138

belief, and knowledge, respectively. Finally, Rosenberg’s (2005) method calculates the number of studies averaging null results that would have to be added to the given set of observed outcomes to reduce significance level (p-value) of the (weighted) average effect size to a target alpha level (i.e., 0.05). Once again, it is important to note that no fail-safe method uses study characteristics in their estimation Becker (2005). Table 4-9 outlines the values of Fail-safe N calculated using each of the three recommended methods.

Table 4-9. Calculated Fail-safe Ns. Method Outcome Attitudes Beliefs Knowledge Rosenthal 6,421 270 18,819 Rosenberg 6,211 406 13,707 Orwin 34 12 467

The Rosenberg method suggests that each of the outcomes would require, at minimum, 406 ‘null’ studies to negate the effect of the results calculated in this study.

Rosenthal’s method at least 270 studies, and Orwin’s method at least 12, which would double the number of studies in the research here. Clearly, knowledge is the least likely to be impacted by bias. Even with the most conservative Fail-safe N calculation, at least

467 null studies would be needed to reflect a change in the significance of the computed effect size from the studies. Therefore, while the review process could have been more robust, a sufficient number of studies was located for each of the outcomes under consideration.

Heterogeneity Estimation

Cooper (2017) explains that homogeneity (or its opposite, heterogeneity) is the formal way to compare the observed variance to that expected from sampling error. To estimate this, the metafor package in R calculates I2 value. I2 the proportion of observed variance reflects the real differences in effect size (Borenstein et al., 2009). Table 4-10 shows I2 for each main variable heterogeneity estimation.

139

Table 4-10. I2 estimates of heterogeneity in outcome variable measures. Variable I2 Attitude 89.07 Beliefs 90.82 Knowledge 94.38

As can be seen by the values in the table, the amount of heterogeneity within the present study is considerable (Cooper, 2017) for all conditions. There are several factors that may play into this. These factors can include differences in groups being measured or the tool used for measurement. The full implications of this heterogeneity on this study and its interpretations are discussed in Chapter 5.

Outcome Variable Results

The primary question under consideration for this study was, “What is the effect of TECTI on second-order barrier constructs (i.e., teacher attitudes, beliefs, and knowledge)?” Studies from between 2007 and 2016 were coded for data. A total of 55 studies were examined with some providing data on multiple outcome variables and/or containing multiple study populations within the study report. Each of these outcomes were analyzed using a random-effects model meta-analysis, and the results are presented here.

Within each outcome variable heading, a forest plot of the results for each outcome variable is provided. The forest plot presents a graphical representation of key study data. The box-and-whiskers display for each study illustrate each study. Each box is centered on the studies effect size, with the relative size being representative of the study’s sample size. The whiskers model the study’s 95% confidence interval (CI). The

140

cumulative effect size of the outcome variable, and its related CI, are illustrated by the diamond figure at the base of the plot.

Attitude

The research question asked: What is the effect of TECTI on teacher attitudes?

The study’s goal was to understand the nature of the effect of the course on attitudes related to technology integration for teaching and learning. The effect on attitude was estimated from the data of thirty-four studies (k = 34) looking at 2,828 preservice teachers (n = 2,828). The estimated effect size on attitude was 0.722 (C.I. 0.497, 0.946), thereby indicating that TECTI have a positive influence on teacher attitudes (see Figure

4-4). This means that on average TECTI can improve attitude scores by 0.722 standard deviations. The corresponding z score indicated the difference in attitude is statistically significant at an α = .05 (z = 6.300, p < 0.0001).

Figure 4-4. Attitude outcome forest plot.

141

Beliefs

The research question asked: What is the effect of TECTI on teacher beliefs? The study’s goal was to understand the nature of the effect of the course on beliefs related to technology integration for teaching and learning. Synthesizing data from 1,094 preservice teachers (n = 1,094) across twelve studies (k = 12), the estimated effect size on beliefs was 0.474 (C.I. 0.031, 0.918), thereby indicating that TECTI have a positive influence on teacher beliefs (see Figure 4-5). Meaning that on average TECTI can improve belief scores by 0.474 standard deviations. The corresponding z score indicated the difference in belief is statistically significant at an α = .05 (z = 2.095, p = 0.036). However, it is important to note that with the wide CI of the summary ES that the impact on beliefs has the potential to be as low as 0.054 standard deviations.

Figure 4-5. Belief outcome forest plot.

142

Knowledge

The research question asked: What is the effect of TECTI on teacher knowledge?

The study’s goal was to understand the nature of the effect of the course on technology knowledge related to technology integration for teaching and learning. This portion of the meta-analysis captured the data from 3,271 preservice teachers (n = 3,271) across 46 studies (k = 46). The estimated effect size on knowledge was 0.854 (C.I. 0.503, 1.204), thereby indicating that TECTI have a positive influence on teacher knowledge (see

Figure 4-6). Meaning that on average TECTI can improve knowledge scores by 0.854 standard deviations. The corresponding z score indicated the difference in knowledge is statistically significant at an α = .05 (z = 4.7750, p < 0.0001).

Figure 4-6. Knowledge outcome forest plot.

Overall, the result of this series of meta-analyses suggests that TECTI have a largely positive effect on preservice teachers’ attitudes, beliefs, and knowledge related to

143

technology integration for teaching and learning. Further interpretation of the impact of the courses and its meaning are discussed in the next chapter.

Sub-group Analyses

Four series of sub-group analyses were conducted for each of the main variables

(i.e., attitudes, beliefs, and knowledge) in this research. The results of each sub-group comparisons are presented in the following subsections.

Course Features

Potential moderating influences on the main effect size consideration in this study are the course features within the given courses. Each main effect was further analyzed base on the presence or absence of a feature as identified by the study report. The eight features considered were: mentoring/coaching, rehearsal/field experience, goal-setting, observation, reflection/self-evaluation, hands-on learning, work sample analysis, and practice lesson planning. A random-effects model was used for these analyses because it was assumed a priori that not all sub-groups would have a common effect size. From there, separate estimates of τ2 were estimated for estimated for each sub-group and used for each group in their analysis. Results are presented by attitude, belief, and knowledge variable.

The attitude variable sub-group analyses examined all eight of the sub-group variables. It is important to note that in the case of goal-setting only a single case of inclusion was identified in the literature. As a result, a fixed-effect model (FEM) was estimate for the group employing goal-setting as a course feature. Through this section, the tables presenting the results report the population for each group (N), the number of included studies (k), the calculated Cohen’s g, the standard error (SE), and the lower and upper 95% confidence interval (CI) boundaries. Additionally, the Q-value, the degrees of

144

freedom (df), and p-value comparing the two sub-groups are presented. Tables 4-11 through 4-18 contain the results of the sub-group analyses for the attitude main effect.

Table 4-11. Comparison of attitude outcome by mentoring/coaching. Study Info ES, SE, and 95% CI N k g SE CI Lower CI Upper Q df p Without 2,370 29 0.748 0.138 0.477 1.018 With 458 5 0.571 0.081 0.411 0.729 Between 0.418 1 0.518

Table 4-12. Comparison of attitude outcome by field experience/rehearsal. Study Info ES, SE, and 95% CI N k g SE CI Lower CI Upper Q df p Without 2,246 30 0.648 0.110 0.432 0.863 With 582 4 0.970 0.232 0.515 1.424 Between 0.757 1 0.384

Table 4-13. Comparison of attitude outcome by goal-setting. Study Info ES, SE, and 95% CI N k g SE CI Lower CI Upper Q df p Without 2,794 33 0.720 0.117 0.491 0.950 With 34 1 0.833 0.253 0.338 1.329 Between 0.043 1 0.835

Table 4-14. Comparison of attitude outcome by observation. Study Info ES, SE, and 95% CI N k g SE CI Lower CI Upper Q df p Without 2,594 30 0.715 0.130 0.461 0.969 With 234 4 0.790 0.092 0.609 0.971 Between 0.061 1 0.805

Table 4-15. Comparison of attitude outcome by reflection/self-evaluation. Study Info ES, SE, and 95% CI N k g SE CI Lower CI Upper Q df p Without 2,232 25 0.782 0.143 0.501 1.062 With 596 9 0.524 0.177 0.177 0.870 Between 1.100 1 0.294

Table 4-16. Comparison of attitude outcome by hands-on learning. Study Info ES, SE, and 95% CI N k g SE CI Lower CI Upper Q df p Without 1,730 21 0.641 0.143 0.360 0.921 With 1,098 13 0.869 0.180 0.517 1.221 Between 1.026 1 0.311

Table 4-17. Comparison of attitude outcome by work sample analysis. Study Info ES, SE, and 95% CI N k g SE CI Lower CI Upper Q df p Without 2,096 27 0.725 0.147 0.432 1.013 With 732 7 0.711 0.155 0.406 1.016 Between 0.003 1 0.953

145

Table 4-18. Comparison of attitude outcome by lesson planning. Study Info ES, SE, and 95% CI N k g SE CI Lower CI Upper Q df p Without 1,531 21 0.600 0.134 0.337 0.864 With 1,297 13 0.859 0.157 0.552 1.166 Between 1.274 1 0.259

In the case of attitude, none of the cases were shown to be statistically significant.

This indicates that presence or absence of the identified course features resulted in minimal or no attitude score differences. The largest differences in favor of including a feature were for hands-on learning and lesson planning. Also, the interpretation of the goal-setting analysis would need further examination, because of the single study with a smaller sample size used for the comparison.

The belief variable sub-group analyses examined only seven of the eight sub- group variables. The included studies yielded no studies where goal-setting was an included feature. As such, no sub-group comparison could be conducted. Additionally, mentoring/coaching, observation, and self-evaluation/reflection only returned single cases of presences in the included literature, thereby resulting in a FEM being used for analysis in those cases. Tables 4-19 through 4-25 contain the results of the sub-group analyses for the beliefs main effect.

Table 4-19. Comparison of beliefs outcome by mentoring/coaching. Study Info ES, SE, and 95% CI N k g SE CI Lower CI Upper Q df p Without 1,010 11 0.508 0.251 0.016 0.999 With 87 1 0.051 0.152 -0.246 0.348 Between 0.616 1 0.433

Table 4-20. Comparison of beliefs outcome by field experience/rehearsal. Study Info ES, SE, and 95% CI N k g SE CI Lower CI Upper Q df p Without 583 10 0.264 0.189 -0.105 0.634 With 511 2 0.691 0.452 -0.195 1.578 Between 0.791 1 0.374

146

Table 4-21. Comparison of beliefs outcome by observation. Study Info ES, SE, and 95% CI N k g SE CI Lower CI Upper Q df p Without 1,018 11 0.479 0.252 -0.016 0.973 With 76 1 0.406 0.164 0.085 0.727 Between 0.015 1 0.902

Table 4-22. Comparison of beliefs outcome by reflection/self-evaluation. Study Info ES, SE, and 95% CI N k g SE CI Lower CI Upper Q df p Without 1,058 11 0.491 0.241 0.019 0.964 With 36 1 -0.009 0.221 -0.411 0.423 Between 0.706 1 0.401

Table 4-23. Comparison of beliefs outcome by hands-on learning. Study Info ES, SE, and 95% CI N k g SE CI Lower CI Upper Q df p Without 1,019 9 0.507 0.239 0.039 0.976 With 75 3 0.034 0.318 -0.589 0.656 Between 1.371 1 0.242

Table 4-24. Comparison of beliefs outcome by work sample analysis. Study Info ES, SE, and 95% CI N k g SE CI Lower CI Upper Q df p Without 982 10 0.497 0.271 -0.034 1.029 With 112 2 0.258 0.209 -0.150 0.667 Between 0.255 1 0.614

Table 4-25. Comparison of beliefs outcome by lesson planning. Study Info ES, SE, and 95% CI N k g SE CI Lower CI Upper Q df p Without 528 8 0.296 0.221 -0.138 0.730 With 566 4 0.625 0.325 -0.013 1.262 Between 0.526 1 0.468

As with attitude, none of the cases were shown to be statistically significant. This indicates that presence or absence of the identified course features resulted in minimal or no beliefs score differences. The most interesting outcome of these analyses is that most of the features had a negative impact on the development of positive technology integration beliefs. However, in most cases there were highly different study counts and population samples, so the conclusions are only indicative of the present primary studies.

The knowledge variable sub-group analyses examined all eight of the eight sub- group variables. As with attitude, knowledge only showed a single case of goal-setting as

147

a course feature, thereby resulting in a FEM for that sub-group. Tables 4-26 through 4-33 contain the results of the sub-group analyses for the knowledge main effect.

Table 4-26. Comparison of knowledge outcome by mentoring/coaching. Study Info ES, SE, and 95% CI N k g SE CI Lower CI Upper Q df p Without 2,958 41 0.865 0.202 0.469 1.260 With 313 5 0.758 0.279 0.212 1.304 Between 0.049 1 0.824

Table 4-27. Comparison of knowledge outcome by field experience/rehearsal. Study Info ES, SE, and 95% CI N k g SE CI Lower CI Upper Q df p Without 2,551 38 0.906 0.195 0.524 1.287 With 720 8 0.699 0.394 -0.073 1.471 Between 0.185 1 0.667

Table 4-28. Comparison of knowledge outcome by goal-setting. Study Info ES, SE, and 95% CI N k g SE CI Lower CI Upper Q df p Without 2,928 45 0.928 0.175 0.585 1.270 With 343 1 0.378 0.069 0.242 0.513 Between 0.551 1 0.458

Table 4-29. Comparison of knowledge outcome by observation. Study Info ES, SE, and 95% CI N k g SE CI Lower CI Upper Q df p Without 3,049 40 0.817 0.154 0.515 1.119 With 222 6 1.518 0.732 0.038 2.953 Between 2.764 1 0.096

Table 4-30. Comparison of knowledge outcome by reflection/self-evaluation. Study Info ES, SE, and 95% CI N k g SE CI Lower CI Upper Q df p Without 2,419 31 0.839 0.239 0.371 1.306 With 852 15 0.894 0.245 0.414 1.374 Between 0.022 1 0.883

Table 4-31. Comparison of knowledge outcome by hands-on learning. Study Info ES, SE, and 95% CI N k g SE CI Lower CI Upper Q df p Without 2,084 31 0.905 0.194 0.524 1.286 With 1,187 15 0.775 0.382 0.025 1.525 Between 0.112 1 0.738

Table 4-32. Comparison of knowledge outcome by work sample analysis. Study Info ES, SE, and 95% CI N k g SE CI Lower CI Upper Q df p Without 2,931 36 0.844 0.172 0.506 1.181 With 340 10 0.953 0.432 0.108 1.799 Between 0.095 1 0.758

148

Table 4-33. Comparison of knowledge outcome by lesson planning. Study Info ES, SE, and 95% CI N k g SE CI Lower CI Upper Q df p Without 1,112 19 0.995 0.219 0.566 1.424 With 2,159 27 0.795 0.220 0.363 1.227 Between 0.410 1 0.522

None of the sub-group analyses for knowledge resulted in statistically significant differences. However, observation did result in a 0.096 p-value in favor of using it. The difference in average effect size was nearly double with an ES of 0.817 for courses without observation to 1.518 for those with observation as a course feature. This was the most significant result across all sub-group analyses related to course features. This may suggest potentially beneficial outcomes from using observation in courses. However, while the number of comparison studies for knowledge were better than the prior two variables, but even still the results should be interpreted carefully.

In summary, none of the eight identified course features resulted in significant differences, either positively or negatively, for any of the main effect variables. This lack of significant results may indicate a couple different conclusions. Those conclusions and their implications are discussed in Chapter 5.

Study Quality Impacts

Multiple measures of study quality were investigated as part of this research. Each general study quality, measurement validity, and reported reliability were used as a sub- group to explore potential differences in study outcome.

Study quality rank

The second sub-question for this study asked: What is the difference in magnitude of effect of TECTI on each second-order barrier construct when comparing levels of overall study quality? As stated last chapter, study quality was calculated through a composite score resulting for multiple data points from the study. Appendices Q, R, and

149

S present the data of study features/design for each variable attitudes, beliefs, and knowledge respectively that contributed to the composite score. Quartile rankings from each outcome were then calculated, and then used to designate the study as low, standard/average, or high quality. Tables 4-34 through 4-36 present the results of each sub-group analysis.

Table 4-34. Comparison of attitude outcome by study quality rank. Study Info ES, SE, and 95% CI N k g SE CI Lower CI Upper Q df p Low 306 5 0.121 0.188 -0.247 0.490 Average 1,812 20 0.764 0.156 0.458 1.070 High 710 9 0.900 0.131 0.643 1.156 Between 7.530 2 0.023

Table 4-35. Comparison of beliefs outcome by study quality rank. Study Info ES, SE, and 95% CI N k g SE CI Lower CI Upper Q df p Low 229 3 0.251 0.453 -0.637 1.139 Average 652 5 0.667 0.308 0.063 1.271 High 213 4 0.091 0.194 -0.289 0.470 Between 1.697 2 0.428

Table 4-36. Comparison of knowledge outcome by study quality rank. Study Info ES, SE, and 95% CI N k g SE CI Lower CI Upper Q df p Low 415 9 1.176 0.368 0.454 1.897 Average 2,322 26 0.837 0.173 0.499 1.175 High 534 11 0.701 0.450 -0.180 1.583 Between 1.280 2 0.527

A couple interesting trends emerge from these results. First, the quality of the study may be impacting the nature of the results for attitude in a statistically significant way. However, as study quality went up, the effect sizes increase. Keeping in mind this differs from the course design, the study design may have some feature (e.g., theoretical foundation, measurement tool, etc.) that may be changing the nature of the effect size.

Second, belief by study quality has the smallest average effect size for the highest quality studies. While not statistically significant, the difference is worth noting. Finally, while not statistically significant, the ES magnitude decreases for the knowledge outcome as

150

study quality increases. While unable to state definitively because of the number out potentially influencing variables, this may suggest that higher quality studies are producing a better approximation of the true ES.

Measurement Validity

The third sub-question for this study asked: What is the difference in magnitude of effect of TECTI on each second-order barrier construct when comparing levels of measurement validity? Validity conditions compared were: study specific tool (SST), previously validated tool with modification (PVM), and previously validated tool without modification (PVWOM). Tables 4-37 through 4-39 present results of each variable by validity level.

Table 4-37. Comparison of attitude outcome by measurement validity level. Study Info ES, SE, and 95% CI N k g SE CI Lower CI Upper Q df p SST 678 4 1.047 0.403 0.257 1.837 PVM 1,119 13 0.542 0.113 0.321 0.764 PVWOM 1,031 17 0.755 0.203 0.358 1.153 Between 2.238 2 0.198

Table 4-38. Comparison of beliefs outcome by measurement validity level. Study Info ES, SE, and 95% CI N k g SE CI Lower CI Upper Q df p SST 186 3 0.643 0.336 -0.015 1.301 PVM 637 5 0.054 0.151 -0.241 0.350 PVWOM 271 4 0.604 0.392 -0.164 1.373 Between 3.101 2 0.212

Table 4-39. Comparison of knowledge outcome by measurement validity level. Study Info ES, SE, and 95% CI N k g SE CI Lower CI Upper Q df p SST 1,697 11 0.752 0.180 0.399 1.106 PVM 996 16 1.122 0.271 0.591 1.653 PVWOM 578 19 0.768 0.218 0.340 1.196 Between 1.543 2 0.462

Overall, there was no statistically difference between the levels of validity for each condition. For both attitude and belief, the modification of a previously validated tool yielded the lowest ES. Conversely, the highest estimated ES for knowledge was in the same condition. Additionally, for both attitude and knowledge a study specific 151

measure yielded the highest ES estimate. This may mean that these studies are impacted by measuring outcomes using a tool that has not been through a vigorous validation process.

Reported Reliability

The third sub-question for this study asked: What is the difference in magnitude of effect of TECTI on each second-order barrier construct when comparing levels of reported reliability? Reliability conditions were determined through the primary studies themselves. This variable related to if the researchers statistically verified the reported reliability of the data they collected regardless of the previously established reliability of the tool. The conditions contrasted were established reliability (E) reported against unestablished reliability (NE) in the report. Reliability was considered established and reported if the Cronbach’s alpha from the primary study was presented by the authors and over 0.75 on average for the data points collected. The results of these analyses are in the following tables.

Table 4-40. Comparison of attitude outcome by reliability. Study Info ES, SE, and 95% CI N k g SE CI Lower CI Upper Q df p NE 1,632 14 0.824 0.177 0.478 1.171 E 1,196 20 0.643 0.133 0.382 0.904 Between 0.560 1 0.454

Table 4-41. Comparison of beliefs outcome by reliability. Study Info ES, SE, and 95% CI N k g SE CI Lower CI Upper Q df p NE 720 7 0.630 0.317 0.008 1.252 E 374 5 0.169 0.264 -0.348 0.686 Between 1.220 1 0.269

Table 4-42. Comparison of knowledge outcome by reliability. Study Info ES, SE, and 95% CI N k g SE CI Lower CI Upper Q df p NE 1,234 23 0.970 0.159 0.659 1.280 E 2,037 23 0.797 0.293 0.222 1.372 Between 0.289 1 0.591

152

Again, the overall statistical effect in each outcome case based on reported reliability were not significant. However, the results are interesting nevertheless. In each case, lowest calculated ES were in those studies that statistically verified the data. This suggests that those studies, wherein the reliability for the primary study was not reported, not verifying the reliability of their data could be, to a degree, presenting spurious results.

Conclusion

This examination of the impact of TECTI on the attitudes, beliefs, and knowledge of preservice teachers netted some interesting results. Each of the outcome variable meta- analyses show TECTI can influence statistically significant results on the second-order beliefs. The sub-group analyses provided insights into the design of courses and the impacts of study quality features. No courses features were shown to make significant changes in the outcome variables. Study quality did not impact the meta-analyses’ outcome for belief and knowledge, but were significant for attitude. Validity and reliability of the measures also were shown to have no significant impact on the effect size estimates in this study. Finally, the heterogeneity of the measurements in this series of meta-analyses were shown to be high, and may be seriously impacting this study.

This chapter has summarized and analyzed the results of both main effect and sub-group meta-analytic processes. An example of data file structure is available in

Appendix J. R code markdown documents for all analyses conducted on attitudes, beliefs, and knowledge are provided in Appendix K, Appendix L, and Appendix M, respectively.

In the next chapter, these results are interpreted in further detail, and the implications, in general and in light of study limitations, are discussed.

153

CHAPTER 5 DISCUSSION

Introduction

The research in this systematic review aimed to quantify and clarify the impact of teacher education courses for technology integration (TECTI) on second-order barriers

(i.e., attitudes, beliefs, and knowledge) inhibiting technology integration The primary research question was: What is the effect of TECTI on second-order barrier constructs

(i.e., teacher attitudes, beliefs, and knowledge)? The related subquestions questions were:

 What is the difference in magnitude of effect of TECTI on each second-order barrier construct when a course feature (i.e., mentoring, observation, rehearsal/field- experience, mentoring/coaching, etc.) is present or absent in a TECTI offering?

 What is the difference in magnitude of effect of TECTI on each second-order barrier construct when comparing levels of overall study quality?

 What is the difference in magnitude of effect of TECTI on each second-order barrier when comparing levels of measure validity?

 What is the difference in magnitude of effect of TECTI on each second-order barrier when comparing levels of reported reliability?

A systematic review of the literature related to teacher education and technology integration was conducted to find studied for analysis that: explored at least one of the main variables, provided empirical measure of that variable, the site of the study was a four- or five-year university-based teacher education program, the course was a stand- alone technology integration course, was published or presented starting in or after 2007, and was in English. Through an iterative process of coding, fifty-five studies were identified in which one or more of the variables was examined. Some studies included multiple populations under examination. Therefore, the final study count of 34 studies on attitudes, 12 studies on beliefs, and 46 studies knowledge examined. Effect sizes were calculated for each study, and a meta-analysis using a random-effects model was run for

154

each main variable. Subsequently, sub-group analyses were used to compare conditions related to course features, study quality, measurement validity, and reported reliability.

These sub-group analyses also employed a random-effects model.

In this chapter, the results of the main effect meta-analyses and relates sub-group analyses are discussed in light of the literature identified through the systematic review, as well as the implications for the findings in terms of the topic as a whole and the course design implications. Furthermore, limitations of the research and conceptualizations of future research studies on the topic are presented.

Main Research Question

The primary focus of the research was to synthesize and statistically analyze the nature of the impact of TECTI on second-order barriers. The study looked to provide a quantified effect of TECTI on each of the second-order barriers, synthesize the literature for each, and investigate the implications. Each second-order barrier constructs’ (i.e., teacher attitudes, beliefs, and knowledge) results, literature discussion, and result’s implications are presented in turn.

Attitudes

In examining the effect of TECTI on teacher attitudes, the results of this study showed that the courses are having a moderate positive effect of the teacher attitudes of preservice teachers as the value of technology for teaching and learning. This result indicates that even a single course on technology integration for preservice teachers can increase attitude measure scores 0.646 standards deviations. Unlike beliefs in the next outcome section, attitudes are relatively consistent across the CI resulting from the meta- analysis. Even at the lower end of the CI, the effect remains moderate and positive, while ranging into a large effect size estimate at the high end of the CI. Therefore, overall the

155

effect on attitudes seems to be consistently significant. A reference list of all articles included in the analysis of attitudes and related sub-groups can be found in Appendix N.

In the research examined through this study, the researchers argue the value of a single course for changing technology integration attitudes. Lee and Lee (2014) state attitude toward computers has been documented by many studies intention and actual practices of technology integration, and teachers with a positive attitude toward computers tend to successfully integrate technology in their classroom. Clearly, this was the contention of this present research, as well. The statistical results show that TECTI positively influence the attitudes of preservice teachers, either by increasing positive attitudes (e.g., self-efficacy) or decreasing negative attitudes (e.g., anxiety). The conclusions by Heo (2009) supported prior research that suggested attitudes change be changed in short period of time, like a single course. However, some researchers suggested that the improvement in preservice teacher attitudes may simple have been the result of taking a course on technology integration, rather than any specific course feature

(Bai & Ertmer, 2008). Milman and Molebash (2008) suggest the significant finding of studies, like the ones under consideration here, may merely represent a “technological high” by preservice teachers experience after completing a technology course.

A large portion of the resulting discussions in the explored studies examine the features of the courses. Alayyar, Fisser, and Voogt (2012) found that attitudes towards

ICT increased through the use of a blended learning environment, but that there were no statistically significant changes in negative attitudes (i.e., anxiety and frustration). Lee and Lee (2014) (2014) credited the change in attitude mostly to practice lesson planning.

Willis (2015) concluded that scaffolding of the learning related to technology integration slowly built confidence, and thereby the significant gains in positive attitudes from their

156

course. Mizell (2016) suggests that specific coaching by the instructor for those students struggling with understanding technology integration helps. Koh and Divaharan (2011) highlighted the positive impact of guided rehearsal. Contrary to these previously mentions studies, Bai and Ertmer (2008) found that some methods from prior research

(e.g., modeling technology use to students) did not necessarily improve technology attitudes. This conclusion more closely mirrors the results of this dissertation. Discussed in detail later this chapter, specific course features did not yield statistical differences in outcomes.

The research on attitudes devoted some studies to the understanding of how different attitude construct and the other outcome variables in this study (i.e., beliefs and knowledge) influence attitudes. Abbitt (2011) found ratings of comfort with computer technology were a significant predictor of self-efficacy beliefs, while perceived usefulness was not. In another study, Abbitt (2011) found positive correlations between self-efficacy and the TPACK domains TPACK, TPK, TCK, TK, and PK. This suggests that increasing knowledge may to a degree increase attitudes positively, as may be expected. Considering these relationships, teacher educators may need to be cognizant of such interplay.

Attitudes, as theorized in this research, are related to the affective positioning of the individual. Outcomes in the affective domain relate to the emotional abilities desired for students (Krathwohl et al., 1964). Of the five affective outcomes identified by

Krathwohl et al. (1964), valuing (i.e., willingness to accept or reject an event through the expression of an attitude) and organizing (i.e., a willingness to organize the values, determine relationships among values, and accept some values as dominant ) were the outcomes most relevant to the current research. The levels of the affective domain form a

157

continuum for attitudinal behavior, from simple awareness and acceptance to internalization, as technology integration attitudes become part of an individual’s practice

(Morrison et al., 2013). The value goal of the teacher education course would be for the preservice teacher to prioritize technology integration. The positive trend in attitudinal change may indicate that preservice teachers are more likely to accept and prioritize technology integration in practice.

Beliefs

In examining the effect of TECTI on teacher beliefs, the results of this study showed that the courses are having a moderate positive effect of the teacher beliefs of preservice teachers as the value of technology for teaching and learning. However, while this impact is statistically significant, the results are highly variable. With a CI ranging from 0.054 to 0.943, the true effect of the courses may be anywhere from negligible to highly effective. This may mean that while educators may be attempting to sway teacher beliefs positively, that the result of the courses may not be as effective as they may hope.

Important factors of this outcome are discussed in the next section. A reference list of all articles included in the analysis of beliefs and related sub-groups can be found in

Appendix O.

While the results of the meta-analysis on the outcome of belief here indicated that

TECTI may have a statistically significant impact on beliefs, the research itself suggest that a single course may have either a negative or positive shift in beliefs. In five of the twelve studies examined (Bai & Ertmer, 2008; C. Clark et al., 2015; Karatas, 2014;

Kounenou, Roussos, Yotsidi, & Tountopoulou, 2015; Lim & Chan, 2007) showed that a single course may have a negative impact on beliefs about the role of technology for teaching and learning. There are several potential reasons for this. Kounenou et al. (2015)

158

suggest students can become overwhelmed with the extent of the training materials that leading to hesitation in putting all technology integration into actual practice. Bai and

Ertmer (2008) reported that the minor change in teacher beliefs in their research were supported by prior studies. They state the deeply rooted nature of beliefs make them difficult to shift through a single course. This was further supported by the correlation of pre-test scores to post-test scores in the measure of beliefs. Bai and Ertmer (2008) suggest that the change to belief happens at key moments when existing beliefs are reflected upon and challenged through the meaningful support and best practice of teacher educators. These conclusions coincide with broader research on belief change. As stated in Chapter 2, Marsh and Wallace (2005) recommend both thought-induced attitude polarization (i.e., visualizing a positive or negative association) and thought introspection

(i.e., reflection on a belief’s root cause) to initiate belief change. It is important to note that when positive change did occur, preservice teachers came to believe that technology increased how the learning is situated (Lim & Chan, 2007). The biggest challenge to this change is the starting point. Bai and Ertmer (2008) found that the positive beliefs of these students for the value of technology for teaching and learning started low. This could be the result of what Ginsburg (2009) called the apprenticeship of observation. This could indicate that stronger focus needs to be place on changing beliefs, because prior examples may have soured the preservice teacher against technology integration.

An aspect of the literature on beliefs developing through TECTI is the strong alignment to technology adoption. Mizell’s (2016) research, and related exploration of prior research, showed that belief in technology’s value for teaching and learning technology was the sole factor, among those considered, directly influencing technology adoption. However, belief was the only factor studied that did not change over time.

159

Anderson’s (2007) research concurs that beliefs have been found to contribute significantly to the prediction future use. Each of these points suggest that while belief may be difficult to shift, it may be the most important potential barrier to remove.

However, it may be moot if, as Cengiz (2015) suggests, intentions to use software will probably vary from their actual behavior in classrooms since the conditions in the schools where they work will likely influence what they ultimately do.

Beliefs, in this study, are teacher pedagogical beliefs related to technology integration. Pajares (1992) suggests that belief change for adults is a rare phenomenon, the most common cause being a conversion from one authority to another or a gestalt shift. This appears to be supported in the research and results. A change in belief may result from effectively reaching the highest level of affective change (i.e., a value complex). Morrison et al. (2013) explain that a value complex means the individual consistently acts in accordance with accepted values and incorporating that behavior as a part of one’s personality (p. 105). Therefore, the result of the technology integration course results in the preservice teacher adopting a technology integration value complex.

While this would be the goal, the results cannot support such a conclusion. The beliefs were significantly shifted in a positive direction, but any predictive conclusion about future practice would be purely speculative at best.

Knowledge

In examining the effect of TECTI on teacher knowledge, the results of this study showed that the courses are having a large positive effect on teacher knowledge related to technology and technology integration. The result indicates that even a single course on technology integration for preservice teachers can increase knowledge measure scores

0.854 standards deviations. However, the CI around this estimate is fairly wide. The span

160

is not enough to decrease the outcome to a level of non-significance, so there is still high level of increased knowledge. Even at the lower end of the CI, the effect remains moderate and positive, while ranging into a very large effect size estimate at the high end of the CI. Therefore, the overall effect on knowledge seems to be consistently significant.

Nevertheless, some researchers argued that a single course was not enough (Lyublinskaya

& Tournaki, 2014). Interpretations of the effect size result from the literature are discussed next. A reference list of all articles included in the analysis of knowledge and related sub-groups can be found in Appendix P.

The literature reviewed made several recommendations for the design of courses related to technology knowledge as defined by this dissertation. Lee and Kim (2014) found that preservice teacher tend to analyze technologies based on external characteristics rather than the affordances that may make the tool valuable in a pedagogical context. Therefore, developing a deeper understanding of technology affordances may be critical for a stronger knowledge of technology integration. Alayyar et al. (2012) found both blended learning and design teams had positive effects on knowledge. Cengiz (2015) recommends a course structure that combines theory lessons with lab session for hands-on and contextual learning. Alexander, Knezek, Christensen,

Tyler-Wood, and Bull (2014), while focusing on project-based learning, concluded that there is no single “magic bullet” that would import knowledge. This conclusion coincides with what this study found related to course features discussed later this chapter.

While the focus of this study was knowledge as an integrated construct, the studies analyzed spoke to the value of separate instruction for each subdomain of

TPACK. Multiple studies highlighted the importance of pedagogical knowledge (PK) in the development of TPACK. Shinas, Karchmer-Klein, Mouza, Yilmaz-Ozden, and J.

161

Glutting (2015) found changes to PK statistically significant changes to TPACK, and recommend a course structure where preservice teachers get to experience practical experience in teaching through field experience may help TPACK development if conducted at the same time as the TECTI. Chai, Koh, and Tsai (2010), via stepwise regression models, identified that PK has the largest impact on TPACK in preservice teachers. This may support the contention by Blankson, Keengwe, and Kyei-Blankson

(2010) that TECTI may be better suited for the end of programs once teaching methodology courses are completed to better prepare the lens of teaching practice prior to the integration of technology.

(TK) or digital literacy. Conceptual knowledge are the skills and knowledge related to both technological pedagogical knowledge, and, to some degree, technological pedagogical content knowledge (TPACK). These knowledge types can be described using dimensions of knowledge from Anderson et al. (i.e., factual, conceptual, procedural, and metacognitive knowledge) or Niess’ dimensions (declarative, procedural, schematic, and strategic knowledge) to define what the focus of knowledge resulting from TECTI should look like. In this study, knowledge was treated as a cohesive unit, so it is impossible to determine which type of knowledge was specifically impacted.

Nevertheless, pressing preservice teacher to greater height of teaching skill and knowledge should be the goal. The results of this study indicate there can be large degree of impact from even a single course. While this cannot predict a change in future

162

practice, the obvious mitigation of the potential knowledge barrier suggests that preservice teachers benefit from focusing strongly on technology integration in a TECTI.

Implications of Main Variable Outcomes

The implications of the main question under examination are generally positive.

Each result shows that teacher education courses for technology integration are positively shifting each attitude, beliefs, and knowledge of preservice teachers. Yet, what does this mean for researchers and educators?

Seemingly, TECTI are positively influencing teacher attitudes related to technology integration. While some researchers suggest that the change of attitude may be a temporary high, it seems viable that through even a single semester course that attitudes can be significantly shifted. Many of the studies under consideration provide some recommendations for potentially useful course features. However, there was no universally consistent method that seemed to be recommended across literature.

Researchers and educators may need to examine more closely what exactly causes the positive shift resulting from the courses, or possibly more importantly whether or not that change is temporary. As research continues, researchers may consider focusing on designing courses for specific attitude change. Understanding how specific attitudes change across a course could better allow for effective course design. Lambert, Gong, and Cuper (2008) conclude that perceived ability has proven to be a significant factor in determining teachers’ and preservice teachers’ levels of use, and that make it important they not be ignored when searching for better ways to prepare future teachers to use technology. The research here would seem to support this assertion. Alternately, investigating the correlational interplay of positive (e.g., self-efficacy) and negative (e.g.,

163

anxiety) attitudes in the area of technology integration would certainly provide foundational knowledge for other course improvements.

The research shows that changes to belief may be hard to enact through a single

TECTI, but that a positive change may be crucial if we hope to get preservice teachers to integrate technology into teaching and learning. The first stage would be an evaluation of technology integration beliefs by the teacher educator and preservice teacher.

Understanding initial beliefs must be central if those beliefs are ever to be challenged and reflected upon. Consistent and ongoing challenges to negative beliefs about the value of technology and its integration would be the next step. These challenges may lead to a positive shift once a preservice teacher comes to better understand those barriers in place causing them to reject the value of technology. Finally, Clark et al. (2015) echoes the thoughts of Mishra and Koehler (2006) placement of TPACK learning in context.

Positive teacher beliefs on technology may result from seeing the positive benefits of those beliefs enacted in context. Considering the limited sample of studies investigated, it is difficult to draw a definitive conclusion about what may be best practice for shift beliefs. However, challenging the beliefs of the preservice teacher and engaging them in a reflection of those beliefs may prove useful. It may simply be a call for teacher educators to be sure to establish beliefs at the beginning of a course, so that those beliefs may, at the very least, be investigated by the preservice teacher.

The present meta-analysis found that even a single course can have a large positive effect on teacher knowledge related to technology and its integration in teaching and learning. Of the three constructs under consideration, knowledge yielded both the most studies contributing data, and the most significant effect. This would seem to suggest that knowledge seems to be the primary focus of TECTI. This flows logically

164

from the expectations of most courses and programs. Continuing with this focus may have benefits beyond knowledge. As was discussed earlier this chapter, a positive change in knowledge can affect a positive change in beliefs and attitudes, as well. However, there did not appear to be a clear consensus across the literature as to how best accomplish knowledge change. Many course features were recommended, but the research here showed that no one is like Alexander et al. (2014) describe a “magic bullet.” Further research is needed to draw more conclusive recommendations on course design. The next best step may be for researchers to explore promoting the appropriate type of knowledge, using either the conceptualization proposed by Anderson et al. or

Niess for the domain, being learned in TECTI. Gronseth et al. (2010) found that student teachers most typically wanted tools education from a TECTI. While technological knowledge is a foundational factor of technology integration, it certainly should not be the singular focus of TECTI. Technology integration involves thinking strategically while planning, organizing, critiquing, and abstracting for specific content, student needs, and classroom situations while concurrently considering the multitude of twenty-first century technologies with the potential for supporting students’ learning (Niess, 2008). It is this strategic thought on the all aspects of technology integration and the operationalizing of that thought that would likely translate into better action in practice. This should be the focus of both teacher educators and educational researchers so to improve teacher education in this domain, and potentially future practice.

Subquestion Interpretations and Implications

Subquestion One - Course Features

The first of the four subquestions explored in this research sought to quantify what, if any, difference in magnitude of effect of TECTI by sub-group related to TECTI

165

features. The sub-group analyses examined the main variable when the study was identified as having one or more of the eight course features. They were mentoring/coaching, rehearsal/field experience, goal-setting, observation, reflection/self- evaluation, hands-on learning, work sample analysis, and practice lesson planning. In every case, there results of the sub-group analyses showed there were no significant differences in the effect sizes resulting from a specific feature. This could be interpreted in one of two ways. This may mean that each feature, having previously been identified in the research as having a benefit, may be equally effective. This is improbable.

Conversely, there may be no one course feature that can improve attitudes, beliefs, or knowledge. This also seems improbable. The broad look at the research may have not fully identified the presence or absence of all course features. These analyses could have been heavily influenced by the primary research reporting. Further research using meta- analysis to investigate the impact of specific course features would probably benefit from the collection of course syllabi to better codify each courses design.

While the research in each of the main variable explore course features, there was not solid consensus on the best way to improve the variables under consideration. This seems to suggest that while teacher educators and the related researchers and research are exploring these topics, that it is possible that we are far from coming up with a universal course structure to positively impact attitudes, beliefs, and knowledge related to technology integration. For knowledge, this may not be an issue, because of the overall large effect of the courses. Conversely, for the construct of beliefs, it may mean that we as researchers need to consider this topic more carefully. The research showed that change to beliefs is hard to enact, and this may coincide with the lack of clear effect by aspects of TECTI.

166

Subquestion Two - Study Quality

The second subquestion looked at the difference, if any, in magnitude of effect of

TECTI on each second-order barrier construct when comparing levels of overall study quality. Unlike the prior sub-group analyses, this research found that study quality did have some impact on the estimation of effect sizes. While neither beliefs or knowledge were shown to have statistically significant differences due to study quality, the estimation of effect sizes for attitudes did show a statistical difference. Yet, the result of this difference appears to be positive. As study quality rank increased, the related effect size also increased.

The interpretation and implication of the quality score results suggest a positive outcome for researchers. The lack of difference in knowledge and beliefs can be interpreted to mean that across all studies that researcher understanding of the constructs being measure is consistent. It is difficult to conclusively say that the understanding is high, but there is possibly a converging understanding of the topic. Alternately, the increase over quality rank in attitude research seems to suggest that those researchers conducting studies at the highest level possess the best understanding of the attitudes related to technology integration. Those studies may potentially be used as benchmarks for future quality research.

Subquestion Three - Validity

Subquestion three explores potential differences in magnitude of effect of TECTI on each second-order barrier when comparing various levels of measurement validity.

Measurement validity was assessed in relation to study specific measures, previously validated measures that were modified for the primary studies evaluated here, and previously validated measures without modification. In each case (i.e., attitudes, beliefs,

167

and knowledge), there were no statistically significant differences. Some trends were interesting in the results. Attitude measure constructed specifically for studies yielded the highest ES estimates for that construct. Meanwhile, study specific measures and previously validated measures for belief were nearly identical in their ES estimates.

Finally, knowledge measures that were modified for the study yielded the highest estimate.

While there were some interesting trends in these results, there cannot be any strong conclusions drawn from the results. It is impossible to say how the differences in measure validity impacted due to the lack of statistical difference. Further research would be needed to understand if there is any potential difference between studies due to measurement validity.

Subquestion Four - Reported Reliability

The final subquestion investigated the possible difference in magnitude of effect of TECTI on each second-order barrier when comparing levels of reported reliability as reported within the primary research. Crocker and Algina (1986) define reliability as the desired consistency, or reproducibility, of test scores. This study compared contrasted groups of studies wherein reliability was established via a statistical measure (e.g.,

Cronbach’s alpha) against those where it was not, as it was reported by the authors. In every case, the reported reliability of the measure had no statistical effect on the estimation of the effect size. However, while there was no statistical significance, in each construct the estimation of effect size was lower when the reported reliability of the measure was established.

As with validity, there is no way to draw a definitive conclusion about the impact of reported reliability on the estimation of the effect size. There is clearly a trend

168

indicating that reported reliability of the measure may be impacting the results. However, it would be improper at this time to say that the reported reliability of the measure impacts the outcomes. Once again, further research would be needed to understand if there is any potential difference between studies due to reliability in any form.

Heterogeneity

True heterogeneity exists among population effect sizes estimated by each study due to the influence characteristics that vary among the studies (Huedo-Medina et al.,

2006). There are various means of assessing heterogeneity in meta-analyses. For this study, an I2 index was calculated for each attitudes, beliefs, and knowledge. As was shown in the last chapter, the percentage of variance due to differences in studies was calculated as 89.07%, 90.82%, and 94.38% for each attitudes, beliefs, and knowledge, respectively. Cooper (2017) defines any amount over 70% as “considerable.” This means that while this research allowed for a quantitative synthesis of the studies examined, any conclusions made from these results must be interpreted vary cautiously.

What may account for the variance between these studies? It is improbable that all factors can be accounted for, but the mostly likely ones are presented. First, as might be expected from such a broad lens look at the literature, the differences in courses are likely to have impacted the outcomes here. Differences in class content, participants, design, focus, etc. all would contribute to a less cohesive conclusion. This was accounted for in the design of the study through the use of the random-effects models, but that in and of itself cannot account for all differences. Second, there are probable differences in the variable. Knowledge had the broadest look with both TPACK, ICT literacy, and 21st-

Century skills being incorporated into a single variable. Attitude looked as efficacy, anxiety, adoption intent, etc. as all components of one construct. Belief had the most

169

unified definition, and even then it did not result in the lowest level of heterogeneity.

Differences in how varying aspects of these constructs were measured were treated as equivalent, but the results here suggest that difference may exist. Finally, some features of the measurement tools may have created differences. For example, Agyei and

Keengwe (2014) examined scatter plots comparing measures of ICT skills and TPACK focused evaluations used in their study. The resulting plots showed strongly linear positive trends in the score comparisons for all TPACK measures. However, each of the

TPACK measures when plotted against the ICT skills measure yielded plots that, while having a positive slope, had substantial dispersion of points around the trend line. This is just one example that summarizes the challenges that may have amplified the heterogeneity in this study. In the future, a more focused study would likely provide more conclusive results.

Limitations and Delimitations

This study was delimited by several study design choices. First, the primary studies included in the research needed to have an empirical comparison of change in a related outcome variable as defined in this meta-analysis. This is a necessity of meta- analytic methods. There are certainly qualitative studies on the topic that provide valuable insights into the topic. Second, participants are preservice teachers participating in a four- or five-year, university-based teacher education program. The systematic review eliminated studies wherein the population of focus was inservice teachers as those studies would have been outside the scope of this research. Thirdly, participants needed to have participated in a stand-alone course on technology integration, as those courses were the focus of this study. Next, all studies must have reported data or indications that allowed calculation or estimation of effect sizes for study effects (e.g. means and

170

standard deviations, t-score, F-score etc.). Some studies were eliminated because the reporting could not make such allowances. Fifth, studies must have been published or presented starting in 2007 and beyond. The systematic review identified some studies that could have proven useful from prior to 2007. However, Mishra and Koehler’s TPACK model heavily influenced the conceptualization of this research. As their original article was published in 2006, 2007 was determined to be the earliest that research studies may include measures of their framework. Next, articles needed to be freely available through the University of Florida library system and its affiliates, and published in English.

Access to the research was crucial to obtain data, and conduct the study. To constrain the scope of the study, these restrictions were enacted. Finally, the search for grey literature involved three delimitations. First, only a forward/backward citation search was employed. Initially contacting researchers in the intersecting domains of technology integration and teacher education was proposed, but to constrain the timeline this was rejected bases on the recommendation of three committee members. Also, the forward search was conducted using Web of ScienceTM. While a robust tool, this may not have located all works referencing a particular article. Finally, if any study yielded more than

25 forward references, the first 25 citations by usage count since 2013 were examined.

There may have been lesser used article with valuable data for this research.

The first limitation of this research potentially stems from the systematic review process. While the process was robust, and yielded an appropriate number of studies, there are several additional measures that may have been taken to improve the overall quality of the review. First, while the search terms and Boolean strings yielded the type of research article needed for this study, additional measures could have been used to better capture the appropriate research. The initial 2,936 studies identified through the

171

database searches contained a number of studies that has no relevance to the final meta- analyses. It is likely that a more refined search string or formula may have better captured the appropriate research. Second, the number of databases queried was selected based on a logical assumption that those databases included research central to the theme of this dissertation. While these databases were certainly appropriate, there may be others that contain relevant research. This research was limited to those databases accessible through the University of Florida library system, and are not fully comprehensive of the full spate of databases available to researchers. Third, a more extensive process of searching for grey literature could have been employed. In the interest of time and management, this research was limited to the database and forward/backward citation searches detailed in

Chapter 3. Additionally, the citation searches had limitations (e.g., capping the forward reviewed article numbers) placed on the process. The addition of Internet searches, personal contacts, research associations requests, or other methods recommended by

Cooper (2017) would certainly extend and bolster the present research. Finally, even with the quality of the systematic review, it is improbable that the present study incorporates all available articles. The Fail-safe N estimates provides in Chapter 4 do suggest that an appropriate number of article were identified for analysis.

Another potential limitation to this research is the combination of effect sizes from studies of differing designs. The literature on meta-analytic practice does suggest that when comparing studies with repeated measures and independent-groups designs that unless it can be shown the alternate designs estimate the same treatment effect, the effect sizes from the two designs should not be combined (Morris & DeShon, 2002). However, when a researcher can provide rational analysis or empirical moderator analyses demonstrating the effect sizes estimate the same effect, then such combination can be

172

justified (Morris & DeShon, 2002). While the differences in design were not run as a separate subgroup for this study, analyses were run to estimate what, if any, impact study design may have had on the outcomes. The results indicated that in each variable

(attitudes, beliefs, and knowledge) that the results between designs were statistically non- significant at α = 0.05 with p-values of 0.428, 0.182, and 0.130 for each variable, respectively. While conducting meta-analyses on each design type separately would likely provide clearer conclusions, these statistical conclusions suggest that the studies may be compared. However, the design differences should be considered when interpreting the results.

A couple limitations are brought about through the design of this study and the resulting sub-group analyses. In this study, a quality score was calculated through the subjective evaluation of 23 factors comprising the scale created for this research. While the scale was created by synthesizing two frameworks developed for establishing study quality in research, a more robust scale may have shown different results. This study assumed that each of the features scored for quality held a similar weight. There is a likelihood that this assumption is not purely true. Additionally, some quality factors (e.g.,

“Did the study authors report on the appropriateness of sample size for impact on statistical quality and/or conduct analyses for power?”) utilized a non-dichotomous scoring, and therefore potentially were credited with a heavier score influence. There are a couple potential options for future studies that may improve the results. First, a rubric scoring system with each factor of study design may improve the nature of the quality scale score. Additionally, a weighting factor by some feature of course and/or study design may balance the scores in a more robust manner. Beyond the design of the sub- group analyses, the number of yielded studies for analysis in multiple cases were limited.

173

In one belief case, the coding yielded no separation of sub-groups, and as such analysis could not be conducted. Furthermore, several cases across the attitudes, beliefs, and knowledge variables garnered only single cases for comparison. Valentine, Pigott, and

Rothstein (2010) show that in meta-analyses with high levels of heterogeneity, such as this research, that the change in power results can vary from approximately 0.2 to 0.6 when compared against moderate and small heterogeneity studies at similar numbers of included studies and effect size estimated. This issue may be further amplified as Hedges and Pigott (2001) explain that in meta-analytic studies that have primary studies with high statistical power will typically yield meta-analyses with high power. This current study suffers from a two-fold problem in that regard. There are relatively few cases, as has been discussed, and many of those primary studies have low statistical power.

Therefore, interpreting and generalizing the results of this study has to be done with very narrow parameters.

Another potential limiting factor of this research is the lack of definitive conclusion of the course effect on attitudes, beliefs, and/or knowledge based on potentially confounding factors. There were multiple facets of educations which simply could not be addressed through this study. One might ask about the confounding factor of the teacher effect. Not all teacher education courses in technology are taught by professional teachers, educational technologists, and/or those who hold a strong pedagogical belief in the value of technology to improve education. Furthermore, some course instructors may view the role of technology integration vastly asked from the role described in this document. In fact, even a cursory scan of the articles included in the meta-analysis showed that teacher effect was not a considered factor in the design of the primary studies. As such, there are several levels of factors related to the teacher

174

questions which require addressing in future research and meta-analyses. In addition to confounding teacher effects, there is potentially a confounding variable in the term technology itself. While this study attempted to look at courses devoted to a popular definition of technology, some course may be more limited in their tool definition.

Additionally, not all courses used a universal platform for the integrated technologies.

While it is hard to imagine that platform issues would impact course outcomes, there was a potential for this. For example, some long time Windows-based PC users find switch to a Mac platform daunting. The added challenge of an alternate platform may introduce a confounding effect on the resulting attitudes, beliefs, and knowledge. One could argue that knowledge must certainly grow from the exposure to the alternate device.

Conversely, some students may find the switch challenging and frustrating, thereby having an inverse effect on any measure of attitude component. Therefore, future meta- analyses may want to consider stricter definitions of program technologies for a clearer understanding of the effect. Finally, not all courses were created equally. Data collected on the nature of the course (i.e., course feature) was limited to the information provided in the articles. Therefore, the lack of statistically significant results in the study may be a function of the primary studies failing to report on features of the course rather than the true absence of those features. This has the potential to be mitigated by direct contact with the researchers for syllabi to more rigorously understand the nature and content of the courses under review.

Related to the final point in the preceding paragraph, interpretation of the reported reliability sub-group analyses should be interpreted carefully. The analyses were based purely on the reported reliability of each study. The lack of report on study specific reliability analyses cannot be interpreted to mean that reliability was not statistically

175

verified by the researchers. There exists the potential that due to limitations on space or decisions in reporting made by the researcher that the reported reliability is not fully representative of the reliability question. It is important to interpret the sub-group analyses’ results as only being representative of those researchers who chose to report those result, and draw further conclusions accordingly.

Additionally, the validity component of this research must similarly be interpreted within the constraints of the study. There are multiple aspects that go into validity (e.g., face validity, construct validity, etc.). While called generally validity, the research here examines only the established validity of the measures within the primary research as reported by the authors. To extend this research beyond that narrow score would misrepresent the nature of the data and analyses.

The final limitation of the research lies in the interpretation of the results. This study was exploratory. The design allows for a snapshot of the impact of TECTI on the second-order barriers under consideration. This is reflected in the heterogeneity among the measurements. The percentage of total between-study variance was 89.07%, 90.82%, and 94.38 for attitudes, beliefs, and knowledge, respectively. Cooper (2017) describes anything above 70% as “considerable heterogeneity.” This means that while the statistical synthesis provide here allows for a clearer understanding of the impact of these courses, there is little predictive value from the research. This research must be purely interpreted for what it provides the reader. That is this research allows for teacher educators and researchers a clearer picture of the impact of TECTI. It provides some further understanding of how courses may be designed to better limit those barriers that prevent teachers from integrating technology. However, there is no predictive outcome from the results that could indicate that TECTI will cause preservice teachers to integrate

176

technology into teaching and learning once they become full profession teachers.

Suggestions for how to extend this study or other lines of research for more conclusive results are provided in the recommendations for further research.

Future Research

There are multiple areas for further research that could stem from this study.

Further looks at each of the variables either concurrently or separately would certainly further the knowledge base on this topic. Presented here are three potential avenues for the next stage of research.

The most logical extension of this present study would be to re-run the study with a more focused framework. A closer examination of any one of the constructs, or the comprising components, would potentially provide a more conclusive outcome. For example, a study in which TPACK is examined domain by domain would deliver researchers a more substantive look at how each domain changes, and potentially identify those course features that provide the best outcomes. Such a study would benefit from a tighter focus on the topic, and potentially by identifying consistent measures of the variables used in such studies. Obviously by unifying the measure, the need for the validity sub-group measure is less relevant, but study quality and reported reliability sub- group analyses could help clarify the picture in each of the studies. Speaking to the quality measure, a more conclusive measure where items are weighted based on likely impact would allow for a better conceptualization of how the study ranks in terms of quality. Additionally, this study may act as a foundation for future meta-analyses on these topics. Researchers conducting the primary studies could be contacted for both unpublished work and course syllabi. The lack of statistically significant results in this study may be the result of a true summary of the impact of the sub-group covariates.

177

However, it may hypothetically be the result of coding based on report information. A clearer understanding of the structure of each course under examination would allow for better coding of course features. The goal of any future meta-analyses would be to aid in clarifying the work begun here.

A natural extension of the research here would be to conduct a comparative meta- analysis of on one or more of the second-order barriers (i.e., attitudes, beliefs, or knowledge) in stand-alone courses against technology integration embedded programs.

Such a study would further clarify the impact of either style of program. Without a comparative piece, the current research lacks an element of conclusive comparison. By conducting a meta-analysis into one or more of the outcome variables explored here in both stand-alone and embedded programs would provide researchers, instructors, and program directors a deeper understanding of the benefits and drawbacks to each method.

Alternatively, single courses from embedded technology integration programs could also be used for contrasts against stand-alone courses. This may potentially allow for a clearer understanding of the magnitude of effect that an embedded course provides, and if there are any differences from stand-alone courses. This could additionally add to the body of knowledge on the best practice for teacher education courses.

Finally, some manner of predictive or longitudinal look at how TECTI impact future behavior would be helpful. Researchers (Hofer & Grandgenett, 2012; Lei, 2009;

Milman & Molebash, 2008) have explored the effects of technology integration courses with longitudinal studies. A stage by stage meta-analytic look at the change over time in studies such as these can provide a deeper look at the topic that looking at single courses cannot. Providing a deeper look into the change may allow for more predictive conclusions. Furthermore, exploring the impact of TECTI as the preservice teachers

178

transition into practice would help educational technologist operating in the domain of teacher education better decide how best practice now impacts teacher practice later.

Conclusion

The intention of this study was to explore the impact of TECTI on second-order barriers influencing the integration of technology by preservice teachers for teaching and learning. Technology integration has been shown to be influenced by beliefs, attitudes, and knowledge in teachers (An & Reigeluth, 2011; Hew & Brush, 2007; Inan & Lowther,

2010; Liu, Ritzhaupt, Dawson, & Barron, 2016; Reid, 2014; Ritzhaupt, Dawson, &

Cavanaugh, 2012). Eliminating or mitigating barriers to integration early in the teacher development cycle may aid in the effective integration of technology PK12. Ginsburg

(2009) defines four stages of teacher development: apprenticeship of observation, preservice, induction, and inservice. TECTI provide a foundational opportunity for teacher educators to influence the direction of technology integration in PK12 classrooms. This study shows that the courses are having a significant effect.

The first question sought to quantify the effect of TECTI on second-order barrier constructs. As was shown in the case of each construct, TECTI have a significant positive impact on each construct. This suggests that even a single course on technology or technology integration can shift the barriers in a positive direction, thereby mitigating the impact of the barrier and potentially increasing the likelihood that a preservice teacher may integrate technology in future practice. However, this conclusion cannot be made conclusively because of the wide-lens focus of this study, and the lack of definitive prediction of future practice.

The second research question sought to identify key course features that impact, either negatively or positively, a positive change in preservice teacher beliefs, attitudes,

179

or knowledge related to technology and technology integration. The results from this study suggest there are no specific course features that result in statistically significant change. This could be interpreted in one of two ways. Either all features are equally effective. This seems improbable. The second is that the broad focus of this meta-analysis series failed to identify specific benefits of course features. A more constrained study may better clarify the questions of course design explored in this research.

The third question looked at how study quality was impacting the estimation of the effect sizes. The results were largely positive, yet only statistically significant in outcome variable of attitudes. However, this could show that researchers are gaining insights into the measurement of these constructs, and this is translating into better studies with more conclusive outcomes.

The last two questions looked at the impact of validity and reliability in the estimation of the true effect. Neither was found to have a significant impact on the estimation. However, there may be more to this story because of the heterogeneity estimated in this research. More studies are needed to draw a clear conclusion.

In order to better prepare future teachers, an understanding of best practice related to technology integration is a necessary component of the research agenda for both educational technologists and teacher educators. However, best practice research in this area has yielded multiple diverging recommendations. This research benefits the literature base through several advantages afforded through the meta-analysis. The primary studies use to draw information typically collected data from single locations or programs. Conversely, this research synthesized a diverse participant population from a worldwide context. As such, the generalizability of the results increases because of the look across demographics. Similarly, this research looked at a sample size not often

180

afforded studies in this area. To quantify, even in the smallest population examined for this research, the sample went from a study with a sample of fourteen to the synthesize sample of 1,094, an increase of over 78 times. The increase in sample size impacts the power of the study, and therefore the interpretability. A further added benefit of this study relates to the contention by Borenstein et al. (2009) that meta-analyses can act as a jumping off point for new research. The quantification of the inconsistencies of the result indicate that there may be between study factors that were unaccounted for in this research. Moreover, the contrast between the significant main effects against the non- significant sub-group analyses could indicate that further large scale investigations into the instructional design practice for TECTI courses or programs may be needed.

Regardless of the statistical significance of the results, a systematic review and the successive series of meta-analyses lent themselves well to the topic under consideration. While a review and synthesis of all existing literature on the topic could certainly add to the conversation, the quantifiable synthesis of the existing data aids in developing a fuller conceptualization of the issue. A takeaway from this current study on the practice of meta-analysis related to the measure of the attitudes, beliefs, and knowledge of preservice teachers is a fuller understanding of the conceptualization of the primary variables. As Crocker and Algina (1986) explain the way a behavior manifests itself must be translatable into a measureable item. While the differences in constructs within this study were assumed to be closely related, whatever the correlational piece that did not exist between, for example, anxiety and self-efficacy only amplified the heterogeneousness in the outcomes. Given the level of heterogeneity in this study, an appropriate critique of this research is that not only could it be comparing apples and oranges, but maybe also bananas and so on to no end. However, as Borenstein and

181

company say, the outcomes are still looking at the fruit. The broad look provides a good stepping stone to the next stage of research, and in that may be the fruitfulness of the study.

Educators, teacher educators, and educational policy makers see the value of technology for teaching and learning. If the value is there, preparing our future teachers for strong pedagogical uses of technology begins with making sure they are fully prepared for the practice. This series of meta-analyses shows teacher educators focusing on technology integration are potentially having a serious positive influence on the likelihood that teachers will integrate technology. Unfortunately, this study is far from conclusive. More research is needed to solidify the conclusions drawn from this research.

However, by taking a broad, exploratory look at the body of exiting research, we may now have a firm stepping off point for the next stage of research studies.

182

APPENDIX A INDEPENDENT SAMPLE DESIGN FORMULAS AND TERMINOLOGY TABLES

Table A-1. Independent sample design formulas. Adapted from Dunst et al. (2004). Formula Required Information/Statistics

M SD SE t χ2 df N r (푀 − 푀 ) 푑 = 퐸 퐶 ⁄ ( 2 2) ✓ ✓ √ 푆퐷퐸 + 푆퐷퐶 ⁄ 2 (푀 − 푀 ) 푑 = 퐸 퐶 ⁄ [ 2 2 ] ✓ ✓ ✓ √ 푆퐷퐸 (푁퐸) + 푆퐷퐶 (푁퐶) 2 푁 + 푁 푑 = 푡√( 퐸 퐶) ✓ ✓ 푁퐸푁퐶 푡(푁 + 푁 ) 푑 = 퐸 퐶 ✓ ✓ ✓ (√푑푓)(√푁퐸푁퐶) 2푡 푑 = √푑푓 ✓ ✓

4χ2 푑 = √ ✓ ✓ 푁 − χ2

Table A-2. Terminology. Adapted from Dunst et al. (2004). Research Literature Term/Symbol Description M, X M Score mean of a dependent or outcome measure SD, sd, S, s, σ SD Standard deviation of a score mean SE, se SE Standard error of a score mean N, n N Sample size or number of participants in a group or study df df Degrees of freedom t t Student’s t-test F F F statistic 2 2 χ χ Chi square statistic r, phi, rho r Correlation d d Cohen’s d effect size

183

APPENDIX B APA META-ANALYSIS CHECKLIST

Title Make it clear that the report describes a research synthesis and include “meta- analysis,” if applicable Footnote funding source(s) Abstract The problem or relation(s) under investigation Study eligibility criteria Type(s) of participants included in primary studies Meta-analysis methods (indicating whether a fixed or random model was used) Main results (including the more important effect sizes and any important moderators of these effect sizes) Conclusions (including limitations) Implications for theory, policy, and/or practice

Introduction Clear statement of the question or relation(s) under investigation: Historical background Theoretical, policy, and/or practical issues related to the question or relation(s) of interest Rationale for the selection and coding of potential moderators and mediators of results Types of study designs used in the primary research, their strengths and weaknesses Types of predictor and outcome measures used, their psychometric characteristics Populations to which the question or relation is relevant Hypotheses, if any Method Inclusion and Operational characteristics of independent (predictor) and dependent (outcome) exclusion variable(s) criteria Eligible participant populations Eligible research design features (e.g., random assignment only, minimal sample size) Time period in which studies needed to be conducted Geographical and/or cultural restrictions Moderator and Definition of all coding categories used to test moderators or mediators of the mediator relation(s) of interest analyses Search Reference and citation databases searched strategies Registries (including prospective registries) searched: Keywords used to enter databases and registries Search software used and version Time period in which studies needed to be conducted, if applicable Other efforts to retrieve all available studies: Listservs queried Contacts made with authors (and how authors were chosen) Reference lists of reports examined Method of addressing reports in languages other than English Process for determining study eligibility: Aspects of reports were examined (i.e, title, abstract, and/or full text) Number and qualifications of relevance judges Indication of agreement How disagreements were resolved Treatment of unpublished studies

184

Coding Number and qualifications of coders (e.g., level of expertise in the area, training) procedures Intercoder reliability or agreement Whether each report was coded by more than one coder and if so, how disagreements were resolved Assessment of study quality: If a quality scale was employed, a description of criteria and the procedures for application If study design features were coded, what these were How missing data were handled Statistical Effect size metric(s): methods Effect sizes calculating formulas (e.g., Ms and SDs, use of univariate F to r transform) Corrections made to effect sizes (e.g., small sample bias, correction for unequal ns) Effect size averaging and/or weighting method(s) How effect size confidence intervals (or standard errors) were calculated How effect size credibility intervals were calculated, if used How studies with more than one effect size were handled Whether fixed and/or random-effects models were used and the model choice justification How heterogeneity in effect sizes was assessed or estimated Ms and SDs for measurement artifacts, if construct-level relationships were the focus Tests and any adjustments for data censoring (e.g., publication bias, selective reporting) Tests for statistical outliers Statistical power of the meta-analysis Statistical programs or software packages used to conduct statistical analyses Results Number of citations examined for relevance List of citations included in the synthesis Number of citations relevant on many but not all inclusion criteria excluded from the meta-analysis Number of exclusions for each exclusion criterion (e.g., effect size could not be calculated), with examples Table giving descriptive information for each included study, including effect size and sample size Assessment of study quality, if any Tables and/or graphic summaries: Overall characteristics of the database (e.g., number of studies with different research designs) Overall effect size estimates, including measures of uncertainty (e.g., confidence and/or credibility intervals) Results of moderator and mediator analyses (analyses of subsets of studies): Number of studies and total sample sizes for each moderator analysis Assessment of interrelations among variables used for moderator and mediator analyses Assessment of bias including possible data censoring Discussion Statement of major findings Consideration of alternative explanations for observed results: Impact of data censoring Generalizability of conclusions: Relevant populations Treatment variations Dependent (outcome) variables Research designs General limitations (including assessment of the quality of studies included) Implications and interpretation for theory, policy, or practice Guidelines for future research

185

APPENDIX C PRISMA CHECKLIST

Section/topic # Checklist item Reported on page # TITLE Title 1 Identify the report as a systematic review, meta- analysis, or both. ABSTRACT Structured 2 Provide a structured summary including, as summary applicable: background; objectives; data sources; study eligibility criteria, participants, and interventions; study appraisal and synthesis methods; results; limitations; conclusions and implications of key findings; systematic review registration number. INTRODUCTION Rationale 3 Describe the rationale for the review in the context of what is already known. Objectives 4 Provide an explicit statement of questions being addressed with reference to participants, interventions, comparisons, outcomes, and study design (PICOS). METHODS Protocol and 5 Indicate if a review protocol exists, if and where it can registration be accessed (e.g., Web address), and, if available, provide registration information including registration number. Eligibility 6 Specify study characteristics (e.g., PICOS, length of criteria follow-up) and report characteristics (e.g., years considered, language, publication status) used as criteria for eligibility, giving rationale. Information 7 Describe all information sources (e.g., databases with sources dates of coverage, contact with study authors to identify additional studies) in the search and date last searched. Search 8 Present full electronic search strategy for at least one database, including any limits used, such that it could be repeated. Study 9 State the process for selecting studies (i.e., screening, selection eligibility, included in systematic review, and, if applicable, included in the meta-analysis). Data 10 Describe method of data extraction from reports (e.g., collection piloted forms, independently, in duplicate) and any process 186

processes for obtaining and confirming data from investigators. Data items 11 List and define all variables for which data were sought (e.g., PICOS, funding sources) and any assumptions and simplifications made. Risk of bias 12 Describe methods used for assessing risk of bias of in individual individual studies (including specification of whether studies this was done at the study or outcome level), and how this information is to be used in any data synthesis. Summary 13 State the principal summary measures (e.g., risk ratio, measures difference in means). Synthesis of 14 Describe the methods of handling data and combining results results of studies, if done, including measures of consistency (e.g., I2) for each meta-analysis.

187

APPENDIX D TITLE AND ABSTRACT CODING PROTOCOL

Block: Meta-Analysis Abstract Coding Tool (7 Questions)

Start of Block: Meta-Analysis Abstract Coding Tool

Q1 Enter the identification code.

______

Q2 Does the title indicate that the research is related to the dissertation topic?

o Yes (1)

o No (2)

o Unsure (3)

Skip To: Q7 If Does the title indicate that the research is related to the dissertation topic? = No

Q3 Does the abstract indicate that the research is quantitative in nature?

o Yes (1)

o No (2)

o Unsure (3)

Skip To: Q7 If Does the abstract indicate that the research is quantitative in nature? = No

188

Q4 Does the abstract indicate that the research focuses on preservice teachers as the target population?

o Yes (1)

o No (2)

o Unsure (3)

Skip To: Q7 If Does the abstract indicate that the research focuses on preservice teachers as the target populat... = No

Q5 Does the abstract indicate that the research site is a technology integration course?

o Yes (1)

o No (2)

o Unsure (3)

Skip To: Q7 If Does the abstract indicate that the research site is a technology integration course? = No

Q6 Does the abstract indicate that the research focuses on attitudes, beliefs, and/or knowledge related to technology integration?

o Yes (1)

o No (2)

o Unsure (3)

Skip To: Q7 If Does the abstract indicate that the research focuses on attitudes, beliefs, and/or knowledge rela... = No

Q7 Should this research be included in the next round of review? If you responded "NO" to any of the previous questions, select "NO" as the response. Otherwise, select "YES".

o Yes (1)

o No (2)

189

End of Block: Meta-Analysis Abstract Coding Tool

190

APPENDIX E SECOND ROUND CODING PROTOCOL

Start of Block: Identification

Q1 Enter the ID code.

______

Q2 Did you gain access to a print or digital version of the study?

o Yes (1)

o No (2)

Skip To: Q3 If Did you gain access to a print or digital version of the study? = Yes Skip To: End of Survey If Did you gain access to a print or digital version of the study? = No

Page Break

191

Q3 Is this a conference proceeding presentation list or book chapter list?

o No (1)

o Yes (2)

Skip To: Q10 If Is this a conference proceeding presentation list or book chapter list? = Yes

Page Break

192

Q4 Does the study examine attitudes, beliefs, and/or knowledge connected to technology integration?

o Yes (1)

o No (2)

Q5 Was the study conducted in a stand-alone course on technology integration for preservice teachers (university undergraduate students or graduate students prior to their first professional placement)?

o Yes (1)

o No (2)

Q6 Does the study provide quantitative data for the calculation of effect size?

o Yes (1)

o No (2)

Q7 Should this study be included in the full coding stage?

o Yes (1)

o No (2)

193

Display This Question: If Should this study be included in the full coding stage? = Yes

Q8 Review the reference list. Are there any study titles that may fit the criteria of this study?

(Study Criteria: 2007 and beyond; Examining technology integration attitudes, beliefs and/or knowledge in preservice teachers)

If a title or abstract indicates possible study inclusion, enter the reference for such study below for later review.

______

Page Break

194

Display This Question: If Should this study be included in the full coding stage? = Yes

Q9 Conduct a forward search in Web of Science. If the study has been cited, examine the first 25 (or less) citation titles and abstracts for potentially qualifying studies. Are there any study titles that may fit the criteria of this study?

(Study Criteria: 2007 and beyond; Examining technology integration attitudes, beliefs and/or knowledge in preservice teachers)

If a title or abstract indicates possible study inclusion, enter reference for such study below for later review.

______

Page Break

195

Display This Question: If Is this a conference proceeding presentation list or book chapter list? = Yes

Q10 Scan presentation titles or chapters in the list. Review abstract if available. Are there any chapter or presentation titles/abstracts that may fit the criteria of this dissertation study?

(Study Criteria: 2007 and beyond; Examining technology integration attitudes, beliefs and/or knowledge in preservice teachers)

If a title or abstract indicates possible study inclusion, enter the title for the study below for later review.

______

End of Block: Identification

196

APPENDIX F DRAFT FULL CODING PROTOCOL

Research Identification R1a. What was the primary author’s last Using this to help create study code. This information will be name? processed by Qualtrics to create an identification code. R1b. What was the year of preparation Using this to help create study code. This information will be or publication of the document? processed by Qualtrics to create an identification code.

R2. Specific study citation. Enter this using APA 6th formatting.

R3. What type of document was this? The available research may include a number of types of research 1 = Journal article documents. Each publication is described as follows: All “journal 2 = Book or book chapter articles” are manuscripts intended for publication in a peer- 3 = Dissertation reviewed journal. Book or book chapters are those documents 4 = MA thesis intended for publication in print or electronic form. Dissertations 5 = Private report are doctoral research. MA thesis are master program culminating 6 = Government report works. Reports are those put together by research organizations 7 = Conference paper (private), or any government agency, regardless of publication 8 = Other (specify) status. Conference reports are any document of presented research. 9 = Undefinable If the document is not one of these types, code it “Other,” and specify the type. If the nature of the document is not fully identifiable, then code it “Undefinable.”

R4. Was this document published? Code for the current publication status. If the work has an 0 = Not published/Undefined identifiable publisher and/or is a report from a government or 1 = Published private agency, it is considered published. Otherwise, code “Not published/Undefined.”

R5. Was this a peer-reviewed document? Any work that has been reviewed for quality control by a panel of 0 = Not peer reviewed/Undefined reviewers. This may include any type of review process (i.e., 1 = Peer reviewed single-blind, double-blind, doctoral committee, etc.).

Construct Under Investigation & Study quality

The following question gauge the nature of the research in the identified document. They are not mutually exclusive, so any study may have all, some, or none of the required second-order barriers under consideration. Please code all that are appropriate.

I1a. Does this study employ measure Attitudes are affective positions related to technology and/or course effects on teacher attitudes? technology integration. This is defined by the measurement tool, 0 = No and not the researcher. Tools designed to measure anxiety, stress, 1 = Yes acceptance, etc. related to technology are considered to measure 2 = Unsure attitude. If unsure, code “Unsure”

I1b. Does this study employ measure Beliefs are evaluative positions related to technology integration course effects on teacher beliefs? in teaching and learning. This is defined by the measurement tool, 0 = No and not the researcher. The pedagogical beliefs a teacher holds 1 = Yes regarding technology integration if verbalized may sound akin to 2 = Unsure “Technology has a value to my students as both learning and 197

practical tool both now and in the future,” the polar opposite of this statement, or something along a continuum between these two extremes.

I1c. Does this study employ measure Knowledge is either practical or conceptual knowledge related to course effects on teacher knowledge? technology or technology integration. Practical knowledge is 0 = No knowledge and/or skills about the operation of technologies, often 1 = Yes names technological knowledge, digital literacy, ICT literacy, etc. Conceptual knowledge is technological pedagogical knowledge (TPK) or technological pedagogical knowledge and content knowledge (TPACK), as explained by (Koehler & Mishra, 2009; Mishra & Koehler, 2006). Either of these would be defined by the measurement tool, and not the researcher. Any tool measuring either or both of these kinds of knowledge falls under this heading.

I1d. Did you answer “Yes” to more than Just clarifying the nature of the study. There may be questions one of the three prior questions? later that require you to comment specifically if the answer here is 0 = No “Yes”. 1 = Yes

If this document does not answer “Yes” or “Unsure” to at least one of Questions I1a-I1c, the coding process can be halted immediately.

I2. Are participants in this study Identify the nature of the research setting. The program should be preservice teachers in a four- or five- at a university, and part of a four- or five-year traditional teacher year university-based teacher education education program. Use your best estimation of this. If the program? research does not specifically indicate that the course is part of a 0 = No nontraditional program, then it can be included in this study. 1 = Yes

I3. Are participants in this study enrolled Identify the nature of the research setting. This article should in a stand-alone technology integration identify a teacher education course for technology integration course? (TECTI). This is any course that is not part of a program where 0 = No technology integration is worked throughout other content classes. 1 = Yes

If this document does not answer “Yes” to Question I2 and/or I3, the coding process can be halted immediately.

I4. What is the research design of the Please define the research type. If the type of design is unclear, study? code “Undefined/Can’t tell” and the primary investigator will 0 = Undefined/Can’t tell follow up later. 1 = Pre-test/Post-test 2 = Experimental 3 = Quasi-experimental 4 = Correlational

I5. What was the participant sample Enter the numeric value for sample size. size?

I6. What second-order barrier(s) does Once again, this is defined by the measurement tool. Identify all the study address? that apply. If you coded “Unsure” to any of Questions I1a-I1c,

198

note that here, and default to the study description of the outcome variable.

I7. Name of measurement tool was used Enter the name or names of the tools used to measure the barrier(s) to measure the outcome? under investigation. If the tool is unnamed, please indicate that. If the tool was a composite of prior tools, please enter all relevant names.

I8. What were page numbers of the Please enter the page number(s) for the statistical outcomes of the reported statistical outcomes and/or research. Enter all relevant information. You do not need to effect size(s)? provide the actual statistics.

I9. Does the study mention course In the description of the study, does the researcher describe the features? design of the course? 0 = No 1 = Yes

I9a. Does this study employ Mentoring/coaching means the specific pairing of the student with mentoring/coaching? a technology proficient teacher/coach other than the course 0 = No/study does not mention course instructor. If you are unsure, please code “Can’t tell/undefined.” features 1 = Yes ? = Can’t tell/undefined

I9b. Does this study employ Rehearsal/field experience means the course contained one or rehearsal/field experience? more activities where the teacher education student interacted with 0 = No/study does not mention course PK12 students and technology for teaching and learning. If you features are unsure, please code “Can’t tell/undefined.” 1 = Yes ? = Can’t tell/undefined

I9c. Does this study employ goal- Goal-setting means setting goals related to increasing proficiency setting? with technology integration. If you are unsure, please code “Can’t 0 = No/study does not mention course tell/undefined.” features 1 = Yes ? = Can’t tell/undefined

I9d. Does this study employ Observation means the preservice teacher observed technology observation? integration practices as technology was being used with PK12 0 = No/study does not mention course students for teaching and learning. If you are unsure, please code features “Can’t tell/undefined.” 1 = Yes ? = Can’t tell/undefined

Q1. Did the study report any sub-group Any sub-group analyses improves the quality of the research. For analyses (i.e., gender, ethnicity, etc.)? this study, we are only interested in if it was provided or not. 0 =No Therefore, this can be coded “Yes” if any sub-group analyses are 1 = Yes present.

Q2. How did the study address potential This means did the researchers adjust for potential biases. If they bias in how the course under did not or failed to comment on this, then this question should be examination was selected? coded “No”. 0 =No

199

1 = Yes

Q3. Did the study report on student This means did the researchers report on students that declined. If participants that declined participation? the researchers failed to comment on this, then this question 0 =No should be coded “No”. 1 = Yes

Q4. Did the study authors report on the This looks at the quality of research sample size. If the sample size appropriateness of sample size for was insufficient or unreported, code this “No.” If defended impact on statistical quality and/or qualitatively, then code, “Yes, sample size was defended.” If conduct analyses for power? defended with power analyses, then code, “Yes, sample size was 0 =Not appropriate/Not reported defended with power analyses.” 1 = Yes, sample size was defended. 2= Yes, sample size was defended with power analyses.

Q5. What kind of sampling was used for This means was each subject randomly assigned to some condition the study? (probability sampling), or were they a sample of convenience or 0 = non-probability sampling purposive (non-probability sampling)? 1 = probability sampling

Q6. Was the nature of the comparison This is going to be a subjective assessment of comparison group group appropriate for the study? quality. If the researchers report that the group was appropriate, 0 = No then this can be coded “Yes.” Additionally, if this used some pre- 1 = Yes /post-test design for the same group, then the comparison group can be considered appropriate. Otherwise, code “No.”

Q7. Was there any management of pre- This is going to be a subjective assessment of pre-course group course differences between comparison differences. If the researchers report that they attempted to manage groups? differences, then this can be coded “Yes.” Additionally, if this 0 = No/Unknown used some pre-/post-test design for the same group, then the 1 = Yes comparison group can be considered appropriate. Otherwise, code “No.”

Q8. Was the randomization of If the researchers report that a feature was randomized, then count participants, instructor, assessor it towards the randomization count. Look for each participants, measuring outcome, and data analyst instructor, assessor measuring outcome, and data analyst present? randomization, and code as appropriate for the total count. 0 = Not present/Not randomized. 1 = One feature was randomized. 2 = More than one feature was randomized.

Q9. Would the course be reproducible If you believe that an experienced instructor could mimic the by others? course, then this may be coded “Yes.” If the course was poorly 0 = Unknown/No designed or the reporting doesn’t provide sufficient description of 1 = Yes. the course, then code “Unknown/No.”

Q10. Was the course design consistent Look to see if in describing the course/study, do the researchers with theory? reference ISTE Standards, TPACK, andragogy, 21st Century 0 = Unknown/No learning, or other reasonably appropriate methodology for 1 = Low instructional design. If unreported or missing some key 2 = High component, code “Unknown/No.” If one or two of these features exist, code “Low.” If more than two are present, code “High.”

Q11. Was the integrity of the course If the course was planned in advance, and maintained that plan maintained? throughout, code “Yes.” Otherwise, code “No.”

200

0 = Unknown/No 1 = Yes.

Q12. Was attrition prevented and/or Did students leave the course? If reported, code “Yes.” Otherwise, reported? code “No.” 0 = Unknown/No 1 = Yes.

Q13. Is the construct validity of Did researchers report on the construct validity of their instrument ascertainable? measurement tool? If reported, code “Yes.” Otherwise, code “No.” 0 = Unknown/No 1 = Yes.

Q14. Was evidence of reliability Did researchers report on the reliability of their measurement tool? reported? If the researchers did not report, then code “Unknown/No.” If the 0 = Unknown/No researchers reported, but the reliability was less than 0.50, then 1 = It was reported, but insufficient. code “It was reported, but insufficient.” If the researchers reported, 2 = It was reported, and sufficient. and the reliability was at or greater than 0.50, then code “It was reported, and sufficient.”

Q15. What was the follow up period When were measurements of study outcomes conducted? Use best between measures? judgement to assign one of the four categories. 0 = Unknown 1 = One class. 2 = Several course weeks 3 = Beginning and end of the course.

Q16. Did the researchers establish If the researchers found that there were assumption violations assumptions of analysis consistent with (e.g., non-normality), did they attempt to correct. If there is no data? discussion of assumptions was presented or the researchers failed 0 = Unknown/No to provide sufficient information about resulting adjustments, code 1 = Established, but inconsistent. No “Unknown/No.” Otherwise, code as appropriate for the provided adjustments. information. 2 = Established, and inconsistent. Adjustments made. 3 = Established, and consistent.

Q17. Did the study report on statistical Conn and Rantz (2003) explain, “A small study with inadequate power? power might be an important source of information even though 0 = Unknown/No findings lacked statistical significance, whereas a large study with 1 = Yes. selection bias might be a less valid source” (p. 326) Therefore, look to see if power was specifically reported, and code appropriately.

201

Q18. Were confounders controlled in This may be coded “Yes” if the report provides any discussion of design addressed in analysis? potential confounders, and indicate that attempts were made to 0 = Unknown/No control for those confounders. 1 = Yes.

Q19. Were exact test statistic values and Values may be considered exact if they are reported to three p levels presented? decimal places. 0 = Unknown/No 1 = Yes.

Q20. Was the need for the research well Look at the research purpose with the critical eye of a peer- stated? reviewer. If the justification for conducting the study is vague, 0 = Unknown/No under-supported and/or just does not make sense to you as a 1 = Yes. reviewer, code “Unknown/No.”

Q21. Was the research grounded in Look at the research conceptual and/or theoretical framework with theory? the critical eye of a peer-reviewer. If the conceptualization of the 0 = Unknown/No study is vague, under-supported and/or just does not make sense to 1 = Yes. you as a reviewer, code “Unknown/No.”

Q22. Was the study clearly described? Look at the research methodology with the critical eye of a peer- 0 = Unknown/No reviewer. If the conceptualization of the study is vague, under- 1 = Yes. supported and/or just does not make sense to you as a reviewer, code “Unknown/No.”

Q23. Did the study state clear research Can you easily and clearly identify the goals of the research? If the questions? question(s) is/are vague and/or just does not make sense to you as 0 = Unknown/No a reviewer, code “Unknown/No.” 1 = Yes.

Q24. Did the study clearly define the Does the study report full demographics for the target population? population? If the researchers fail to report pertinent demographic information 0 = Unknown/No (at minimum age, gender, and ethnicity) of the participants, code 1 = Yes. “Unknown/No.”

Q25. Did the study define technologies If the study mentions the technology focuses of the course, code related to the course? “Yes.” Otherwise, code “Unknown/No.” 0 = Unknown/No 1 = Yes.

Coder Information C1. What is your coder ID number? Enter your assigned ID.

202

C2. On what date did you complete Enter the coding date. coding this study?

C3. In minutes, how long did it take to This will be recorded by the Qualtrics system. code this study?

203

APPENDIX G FULL CODING PROTOCOL

Start of Block: Default Question Block

Q1 Enter the article's ID Code.

______

Page Break

204

Q2 Was this article published?

o Yes (1)

o No (0)

Q3 Was this document peer-reviewed?

o Not peer-reviewed/Unknown (0)

o Peer-reviewed (1)

Page Break

205

Q4 Identify the focus of the study.

▢ Attitudes about technology and/or technology integration. (1)

▢ Beliefs about integrating technology into teaching and learning. (2)

▢ Knowledge of technology and/or technology integration for teaching and learning. (3)

▢ This article does not address topics related to this meta-analysis. (4)

Skip To: End of Survey If Identify the focus of the study. = This article does not address topics related to this meta-analysis.

Page Break

206

Q5 What is the research design of the study?

o Undefined/Can't Tell (1)

o Pre-test/Post-test (2)

o Experimental Pre-test/Post-test (3)

o Experimental (4)

o Quasi-Experimental (5)

o Correlational (6)

o The design of this study is not appropriate for the current meta-analysis. (7)

Skip To: End of Survey If What is the research design of the study? = The design of this study is not appropriate for the current meta-analysis.

Q6 How many credits did the course consist of?

▼ Unknown (0) ... More than 5 (6)

Q7 Enter the participant sample size(s).

______

Q8 Enter the name/designation of the tool(s) used for measurement.

______

Q9 Enter the page(s) where effect size calculation data or effect sizes can be found.

______207

Page Break

208

Q10 Does the study describe course features?

o Yes (1)

o No (2)

Skip To: Q12 If Does the study describe course features? = No

Q11 Did the study described use, any of the following?

▢ Mentoring/Coaching (1)

▢ Rehearsal/Field Experience (2)

▢ Goal-Setting (3)

▢ Observation (4)

▢ Reflection/Self_Evaluation (5)

▢ Hands-On Learning (6)

▢ Work Sample Analysis (7)

▢ Practice Lesson Planning (8)

▢ Any special course features not listed above. (9)

Display This Question: If Did the study described use, any of the following? = Any special course features not listed above.

Q12 Describe the special course feature/design identified in the previous question.

______

Page Break

209

Q13 Was the need for the research well stated?

o Yes (1)

o No (0)

Q14 Was the research grounded in theory?

o Yes (1)

o No (0)

Q15 Was the research clearly described?

o Yes (1)

o No (0)

Q16 Was the research clearly define the population?

o Yes (1)

o No (0)

210

Q17 Did the study report any sub-group analyses (i.e., gender, ethnicity, etc.)?

o Yes (1)

o No (0)

Q18 Did the study address potential bias in how the course under examination was selected?

o Yes (1)

o No (0)

Q19 Did the study report on student participants that declined participation?

o Yes (1)

o No (0)

Q20 Did the study authors report on the appropriateness of sample size for impact on statistical quality and/or conduct analyses for power?

o Not appropriate/Not reported (0)

o Yes, sample size was defended. (1)

o Yes, sample size was defended with power analyses. (2)

211

Q21 What kind of sampling was used for the study?

o Non-probability sampling (0)

o Probability sampling (1)

Q22 Was a comparison group used and appropriate for the study?

o No comparison group. (0)

o Comparison group inappropriate. (1)

o Comparison group appropriate. (2)

Q23 Was there any management of pre-course differences between comparison groups?

o Pre-course differences are unaddressed. (0)

o Pre-course differences are addressed. (1)

Q24 Did the researchers randomize participants, instructor, assessor measuring outcome, and data analyst?

o Not reported/Not randomized/Unknown (0)

o One feature was randomized. (1)

o More than one feature was randomized. (2)

212

Q25 Would the course be replicable by others?

o Not replicable based on this report. (0)

o Some replicability based on this report. (1)

o Highly replicable based on this report. (2)

Q26 To what extent was the course design consistent with theory?

o Not/Unknown (0)

o Low (1)

o High (2)

Q27 Was the integrity of the course maintained?

o No/Unknown (0)

o Maybe (1)

o Yes (2)

Q28 Did the study define technologies related to the course?

o No (0)

o Vaguely described. (1)

o Described in detail. (2)

213

Q28 Was study attrition prevented and/or reported?

o Not reported (0)

o Not prevented (1)

o Reported and prevented (2)

Q30 What was the measurement period between measures?

o Unknown (0)

o One class period (1)

o Several class periods/weeks (2)

o Beginning and end of course (3)

Q31 Did the researchers establish assumptions of analysis consistent with data?

o Unknown/No (0)

o Established, but inconsistent. No adjustments. (1)

o Established, and inconsistent. Adjustments made. (2)

o Established, and consistent. (3)

214

Q32 Were confounders controlled in design addressed in analysis?

o No/Unknown (0)

o Yes (1)

Q33 Were exact test statistic values and p levels presented?

o No/Unknown (0)

o Yes (1)

End of Block: Default Question Block

Start of Block: Block 1

Q34 Did the researcher establish the reliability of the measure used for the study?

o No/Unknown (1)

o Established, but not reliable. (2)

o Established and reliable. (3)

Q35 Does the study employ a self-reported measure of the construct?

o Yes (1)

o No (0)

215

Q36 What best describes the measurement tool of the study?

o Study specific tool (0)

o Previously validated tool with modification (1)

o Previously validated tool without modification (2)

End of Block: Block 1

216

APPENDIX H FINALIZED CODEBOOK

Research Identification Q1. Enter the article's ID Code. Enter the assigned identification code that matches the study document. Q2. Was this article published? A study can be considered published if it was formally released by a publishing agency (e.g., journal, book, conference, etc. publisher). Yes (1) Studies that are not published are coded “No.” No (0)

Q3. Was this document peer-reviewed? A study can be considered peer-reviewed if it went through a blind review process. Dissertations are not peer-reviewed. Not peer-reviewed/Unknown (0) Peer- If a determination cannot be made, code the study “Unknown.” reviewed (1)

Construct Under Investigation & Course Features

Q4. Identify the focus of the study. This check-box question is to identify which construct(s) the study examines. Attitudes are affective positions related to technology and/or Attitudes about technology and/or technology integration. This is defined by the measurement tool, and not technology integration. (1) the researcher. Tools designed to measure anxiety, stress, acceptance, Beliefs about integrating technology into etc. related to technology are considered to measure attitude. Beliefs are teaching and learning. (2) evaluative positions related to technology integration in teaching and Knowledge of technology and/or technology learning. This is defined by the measurement tool, and not the integration for teaching and learning. (3) researcher. The pedagogical beliefs a teacher holds regarding technology This article does not address topics related integration if verbalized may sound akin to “Technology has a value to to this meta-analysis. (4) my students as both learning and practical tool both now and in the future,” the polar opposite of this statement, or something along a continuum between these two extremes. Knowledge is either practical or conceptual knowledge related to technology or technology integration. Practical knowledge is knowledge and/or skills about the operation of technologies, often names technological knowledge, digital literacy, ICT literacy, etc. Conceptual knowledge is technological pedagogical knowledge (TPK) or technological pedagogical knowledge and content knowledge (TPACK), as explained by (Koehler & Mishra, 2009; Mishra & Koehler, 2006). Either of these would be defined by the measurement tool, and not the researcher. Any tool measuring either or both of these kinds of knowledge falls under this heading. Any study unable to fall into one or more of these three categories are to be coded as not addressing topics related to this meta-analysis study.

If Q4 determines that the study does not address a main effect variable of the study, the coding process will be halted immediately through programming logic.

Q5. What is the research design of the Code the study for research design as defined by the study author(s). For study? any study wherein the study design is not specifically clear, code the Undefined/Can't Tell (1) study “Undefined/Can't Tell” and continue with the coding process. Pre-test/Post-test (2) Such studies will be reviewed and recoded for study design after coding Experimental Pre-test/Post-test (3) is complete. If the study is found to be inappropriate for consideration in Experimental (4) this research, code it as such. This may also include population focus. Quasi-Experimental (5) Correlational (6) The design of this study is not appropriate for the current meta-analysis. (7)

217

If Q5 determines that the study design is not appropriate for this research, the coding process will be halted immediately through programming logic.

Q6. How many credits did the course consist Identify the number of credits the course awarded. If unavailable, try to of? estimate based on hours per week or other indicator. If unable to make a determination, code “Unknown.” Unknown (0) 1 (1) 2 (2) 3 (3) 4 (4) 5 (5) More than 5 (6)

Q7. Enter the participant sample size(s). Enter the numeric value of the sample size, or the page number where those numbers are located. Q8. Enter the name/designation of the Enter the names of all tools. tool(s) used for measurement. Q9. Enter the page(s) where effect size Enter the page numbers. Data will be extracted in a separate step. calculation data or effect sizes can be found. Q10. Does the study describe course Course features are mentoring/coaching, rehearsal/field experience, features? goal-setting, observation, reflection/self-evaluation, hands-on learning, work sample analysis, and practice lesson planning. Each feature is Yes (1) described below. For any study not describing features, code “No.” No (2)

Q11. Did the study described use, any of the If features are described in the study, identify which specific features following? were used in the courses. Mentoring/coaching is any course that the Mentoring/Coaching (1) active encouragement and support of a technology integrating teacher Rehearsal/Field Experience (2) was provided. Rehearsal/field experience is the implementation of Goal-Setting (3) technology integrated lessons in practical PK12 settings. Goal-setting Observation (4) means setting of specific goals for the improvement of technology Reflection/Self_Evaluation (5) integration practice by the preservice teacher. Observation is observing a Hands-On Learning (6) practicing PK12 inservice teacher utilize technology for teaching and Work Sample Analysis (7) learning. Reflection/self-evaluation means preservice teachers reflected Practice Lesson Planning (8) on or self-evaluated personal practice or work samples for the betterment Any special course features not listed above. of technology integration practice. Hands-on learning means the (9) examination of technology tools for how they operate and may be applied in the PK12 classroom. Work sample analysis means the analysis or critique of resources or lessons created by inservice teachers that integrate technology. Practice lesson planning means the student was required to apply knowledge through the design of a technology integrated lesson. If you identify any feature not fulfilled by one of these descriptions, then code “Any special course features not listed above.”

If you identify any previously unidentified course feature, programming logic will prompt you to describe the feature. Otherwise, the system will skip Q12 and move directly to Q13 Q12. Describe the special course Provide a description of the feature. feature/design identified in the previous question.

218

Study quality Features Q13. Was the need for the research well Research may be considered well-stated if the researcher presented a stated? solid argument in favor of the study. The argument must be thoughtful, thorough, and supported by literature. This assessment will be largely Yes (1) subjective, and will require you to interpret their argument as best you No (0) can.

Q14. Was the research grounded in theory? Research may be considered grounded in theory if the research provides a theoretical and/or conceptual framework that is thoughtful, thorough, Yes (1) and supported by literature. This assessment will be largely subjective, No (0) and will require you to interpret their research grounding as best you can. Q15. Was the research clearly described? The research may be considered clearly described if the description can be followed without confusion and the need for clarity. If you find Yes (1) yourself asking question about any aspect of the research, it may not be No (0) clearly described. This assessment will be largely subjective, but use your best judgement as an academic to make a determination.

Q16. Was the research clearly define the Research clearly defines the population if the study provides information population? (Edit: Does the research clearly on the participants that goes beyond a basic description. This may define the population) include a definition of gender, age, academic year, etc. This assessment will be largely subjective, and will require you determine if you believe Yes (1) the researchers should have spoken more to the nature of the No (0) participants.

Q17. Did the study report any sub-group Explore if the researchers conducted sub-group analyses. These will analyses (i.e., gender, ethnicity, etc.)? typically be post-hoc analyses based on sub-groups. If the researchers even include a single sub-group analysis, this may be coded “Yes.” Yes (1) No (0)

Q18. Did the study address potential bias in Bias from course selection would include things like the researcher how the course under examination was being the instructor or convenience. While bias should be avoided, selected? acknowledgement of such bias improves the interpretability of the study.

Yes (1) No (0)

Q19. Did the study report on student This is coded “Yes” if the author reported on students declining participants that declined participation? participation. This may need to be interpreted from differences in total population and participating population. Yes (1) No (0)

Q20. Did the study authors report on the If the author fails to report or acknowledges the sample size is appropriateness of sample size for impact on inappropriate in any way, use the first coding option. If the sample size statistical quality and/or conduct analyses was defended qualitatively, use the second option. If the researchers for power? conducted and reported power analyses, use the third option.

Not appropriate/Not reported (0) Yes, sample size was defended. (1) Yes, sample size was defended with power analyses. (2)

219

Q21. What kind of sampling was used for Code as appropriate for the study as reported by the author. the study?

Non-probability sampling (0) Probability sampling (1)

Q22. Was a comparison group used and If the study used a design that didn’t employ the use of a comparison appropriate for the study? group, code with the first option. For the second option, make a determination from the report about the appropriateness of the No comparison group. (0) comparison group. If in your subjective determination or through author Comparison group inappropriate. (1) reporting it is decided that the group was inappropriate, used the second Comparison group appropriate. (2) coding option. Otherwise, code as appropriate.

Q23. Was there any management of pre- In the study report, if the authors describe pre-course differences and course differences between comparison attempt to control the study in any manner for those differences, then groups? differences may be considered addressed and coded as such. Otherwise, use the first option. Use your best determination as an academic. Pre-course differences are unaddressed. (0) Pre-course differences are addressed. (1)

Q24. Did the researchers randomize Look for discussion of methods in the research report. If you find no participants, instructor, assessor measuring indication or the author specifically comments on the lack of randomize outcome, and data analyst? features, the use the first code. Otherwise, code as appropriate based on the report description. Not reported/Not randomized/Unknown (0) One feature was randomized. (1) More than one feature was randomized. (2)

Q25. Would the course be replicable by This is speaking directly about the design of the course used as the others? context of the study. If the course is not described, use the first coding option. If the course is described to a minimal degree (e.g., provides Not replicable based on this report. (0) some explanation of course design, but less than a week-by-week Some replicability based on this report. (1) breakdown of the course), use the second option. If the report provides a Highly replicable based on this report. (2) reasonably detailed description of the course that could be used as a framework for course design, use the third coding option. Use your best subjective determination. Q26. To what extent was the course design Look to see if in describing the course/study, do the researchers consistent with theory? reference ISTE Standards, TPACK, andragogy, 21st Century learning, or other reasonably appropriate methodology for instructional design. If Not/Unknown (0) there is no way to determine the theoretical influences of the current Low (1) body of research, use the first option. If the author speaks some to the High (2) research and how it relates to the design of the course, use the second option. If the author supports the design of the course through existing literature and the argument is sound, use the third option. Use your best subjective determination.

Q27. Was the integrity of the course If the course design changes were made during the implementation of maintained? the course or the report fails to provide enough description to make a determination, use the first option. If the course seems to use the same No/Unknown (0) design throughout without modification yet the report makes you stretch Maybe (1) your interpretation of this, use the second option. If they design a course Yes (2) and use the same design without modification through the full implementation of the course, use the third option. Use your best subjective determination.

220

Q28. Did the study define technologies If the study report provides no description of what the technology focus related to the course? is within the TECTI, use the first option. If the report names some technologies used in the course and/or vaguely describes technologies No (0) used in the course, use the second option. If you get a good sense of the Vaguely described. (1) technological focus of the course, use the third option. Use your best Described in detail. (2) subjective determination.

Q29. Was study attrition prevented and/or Did all participants that begin the study complete the study? This may be reported? stated by the author by reporting on attrition and/or determined by contrasting starting participant counts against final participant counts. If Not reported (0) attrition is not reported, use the first option. If the study had people Not prevented (1) leave, use the second option. If all participants that started the study Reported and prevented (2) completed the study, use option three.

Q30. What was the measurement period Report the measurement periods as best can be extracted from the study between measures? report. Code as appropriate.

Unknown (0) One class period (1) Several class periods/weeks (2) Beginning and end of course (3)

Q31. Did the researchers establish This relates to assumptions of the data collection and analysis. If the assumptions of analysis consistent with report doesn’t speak to this, use the first option. If the authors speak to data? the assumptions and those assumptions are not met resulting in study design adjustments, use the second option. If the assumptions are not Unknown/No (0) met but the researchers make adjustments to account for this, use the Established, but inconsistent. No third option. If the assumptions are checked and met, use the final adjustments. (1) option. Established, and inconsistent. Adjustments made. (2) Established, and consistent. (3)

Q32. Were confounders controlled in design If the researchers do not speak to potential confounders to their addressed in analysis? methods/variables or a determination can’t be made from the report, use the first option. Otherwise, use the second option. No/Unknown (0) Yes (1)

Q33. Were exact test statistic values and p Values are considered exact if they are reported to the third decimal levels presented? place in most or all cases. In such cases, code “Yes.” In other cases, code “No.” No/Unknown (0) Yes (1)

Q34. Did the researcher establish the If the researcher does not report on the reliability of the measure, use the reliability of the measure used for the study? first option. If the researchers are using a measure that is unreliable (r < .50), use the second option. If the researchers are using a measure that is No/Unknown (1) established either quantitatively or qualitatively as reliable, use the third Established, but not reliable. (2) option. Established and reliable. (3)

Q35 Does the study employ a self-reported If the study reports that the measure used requires the participant to self- measure of the construct? assess via a Likert scale or other method, use the first option. If the measure in not self-reported, use the second option. Yes (1)

221

No (0)

Q36. What best describes the measurement If the tool measuring the variable under consideration was specifically tool of the study? designed for the study, use the first option. If the tool measuring the variable under consideration was modified from a previously validate Study specific tool (0) tool, use the second option. Modification may be translation, rewording Previously validated tool with modification of the question, using only a portion of the measure, or other method that (1) changes the tool items in some way. If the tool measuring the variable Previously validated tool without under consideration was previously validated and the primary study modification (2) researchers made no changes to the tool, then use the third option.

222

APPENDIX I PRISMA FLOWCHART

223

APPENDIX J DATA EXAMPLE

Provided is an MS Excel Worksheet Object showing a sample of the data file setup for the belief variable. Article authors have been redacted to protect the data in anticipation of future publication. id n es var wt mentor_1 fexp_2 goals_3 observ_4 reflect_5 hands_6 wsan_7 lps_8 q_score qs_rank reliable valid CM33 78 1.14373716 0.02983377 33.5190681 0 0 0 0 0 0 0 0 10 low 1 SST ON17 36 -0.0089694 0.04861209 20.5710148 0 0 0 0 1 0 1 1 18 high 1 SST AJ73 109 -0.2681056 0.01543394 64.7922737 0 0 0 0 0 0 0 0 10 low 1 PVWOM EZ23 75 -0.0437176 0.02239908 44.6446987 0 0 0 0 0 0 0 0 17 ave 1 PVM EY32 76 0.40598704 0.02685798 37.2328834 0 0 0 1 0 0 1 0 21 high 1 PVM HS41 72 0.53771091 0.03109678 32.1576745 0 0 0 0 0 0 0 0 15 ave 0 SST DG28 42 0.43175062 0.04872862 20.5218194 0 0 0 0 0 1 0 0 12 low 0 PVWOM FD11 424 0.8033871 0.00402435 248.487086 0 1 0 0 0 0 0 1 16 ave 0 PVWOM HK30 62 0.98534468 0.036173 27.6449278 0 0 0 0 0 0 0 0 14 ave 0 PVWOM BW93 19 -0.0572424 0.10530627 9.49611055 0 0 0 0 0 1 0 1 16 ave 0 PVM HS60 14 -0.592233 0.08568197 11.6710671 0 0 0 0 0 1 0 0 18 high 0 PVM JB35 87 0.05104128 0.02299599 43.4858388 1 1 0 0 0 0 0 1 23 high 0 PVM

224

APPENDIX K R CODE MARKDOWN DOCUMENT - ATTITUDE

Data Summary, Main Effect Analyses, and Model Fit Comparison

#read data Afinal <- read.csv("ATT_final2.csv") summary(Afinal)

## study id es ## Abbitt (2011) : 1 AJ73 : 1 Min. :-0.2681 ## Abbitt & Klett (2007.1): 1 BY76 : 1 1st Qu.: 0.2877 ## Abbitt & Klett (2007.2): 1 CB73 : 1 Median : 0.6127 ## Abbitt & Klett (2007.3): 1 CM33 : 1 Mean : 0.6433 ## Abbitt & Klett (2007.4): 1 DG28 : 1 3rd Qu.: 0.9543 ## Agyei & Voogt (2015) : 1 DU60 : 1 Max. : 1.9761 ## (Other) :28 (Other):28 ## var wt reliable valid ## Min. :0.004382 Min. : 8.646 Min. :0.0000 PVM :13 ## 1st Qu.:0.019183 1st Qu.: 13.495 1st Qu.:0.0000 PVWOM:17 ## Median :0.037893 Median : 27.077 Median :1.0000 SST : 4 ## Mean :0.045762 Mean : 39.863 Mean :0.5882 ## 3rd Qu.:0.074259 3rd Qu.: 52.172 3rd Qu.:1.0000 ## Max. :0.115661 Max. :228.210 Max. :1.0000 ## ## self_report mentor_1 fexp_2 goals_3 ## Min. :0.0000 Min. :0.0000 Min. :0.0000 Min. :0.00000 ## 1st Qu.:1.0000 1st Qu.:0.0000 1st Qu.:0.0000 1st Qu.:0.00000 ## Median :1.0000 Median :0.0000 Median :0.0000 Median :0.00000 ## Mean :0.9706 Mean :0.1471 Mean :0.1176 Mean :0.02941 ## 3rd Qu.:1.0000 3rd Qu.:0.0000 3rd Qu.:0.0000 3rd Qu.:0.00000 ## Max. :1.0000 Max. :1.0000 Max. :1.0000 Max. :1.00000 ## ## observ_4 reflect_5 hands_6 wsan_7 ## Min. :0.0000 Min. :0.0000 Min. :0.0000 Min. :0.0000 ## 1st Qu.:0.0000 1st Qu.:0.0000 1st Qu.:0.0000 1st Qu.:0.0000 ## Median :0.0000 Median :0.0000 Median :0.0000 Median :0.0000 ## Mean :0.1176 Mean :0.2647 Mean :0.3824 Mean :0.2059 ## 3rd Qu.:0.0000 3rd Qu.:0.7500 3rd Qu.:1.0000 3rd Qu.:0.0000 ## Max. :1.0000 Max. :1.0000 Max. :1.0000 Max. :1.0000 ## ## lps_8 q_score qs_rank ## Min. :0.0000 Min. : 9.00 ave :20 ## 1st Qu.:0.0000 1st Qu.:13.00 high: 9 ## Median :0.0000 Median :16.50 low : 5 ## Mean :0.3824 Mean :16.29 ## 3rd Qu.:1.0000 3rd Qu.:19.75 ## Max. :1.0000 Max. :26.00 ##

225

#Fixed-Effect Modeling library(metafor)

## Warning: package 'metafor' was built under R version 3.4.1

## Loading required package: Matrix

## Warning: package 'Matrix' was built under R version 3.4.2

## Loading 'metafor' package (version 2.0-0). For an overview ## and introduction to the package please type: help(metafor). model1<-rma(measure="SMD", yi=es, vi=var, weights=wt, method="FE", data=Afina l, slab=paste(study))

summary(model1)

## ## Fixed-Effects Model (k = 34) ## ## logLik deviance AIC BIC AICc ## -175.7374 403.0265 353.4748 355.0011 353.5998 ## ## Test for Heterogeneity: ## Q(df = 33) = 403.0265, p-val < .0001 ## ## Model Results: ## ## estimate se zval pval ci.lb ci.ub ## 0.7215 0.0272 26.5626 <.0001 0.6683 0.7748 *** ## ## --- ## Signif. codes: 0 '***' 0.001 '**' 0.01 '*' 0.05 '.' 0.1 ' ' 1

#Random-Effects Modeling model2<-rma(measure="SMD", yi=es, vi=var, weights=wt, method="REML", data=Afi nal,slab=paste(study)) summary(model2)

## ## Random-Effects Model (k = 34; tau^2 estimator: REML) ## ## logLik deviance AIC BIC AICc ## -24.0018 48.0037 52.0037 54.9967 52.4037 ## ## tau^2 (estimated amount of total heterogeneity): 0.2108 (SE = 0.0619) ## tau (square root of estimated tau^2 value): 0.4592 ## I^2 (total heterogeneity / total variability): 89.07%

226

## H^2 (total variability / sampling variability): 9.15 ## ## Test for Heterogeneity: ## Q(df = 33) = 403.0265, p-val < .0001 ## ## Model Results: ## ## estimate se zval pval ci.lb ci.ub ## 0.7215 0.1145 6.3002 <.0001 0.4971 0.9460 *** ## ## --- ## Signif. codes: 0 '***' 0.001 '**' 0.01 '*' 0.05 '.' 0.1 ' ' 1

#Model Fit Assessment anova(model2, model1)

## df AIC BIC AICc logLik LRT pval QE ## Full 2 52.0037 54.9967 52.4037 -24.0018 403.0265 ## Reduced 1 355.3223 356.8188 355.4514 -176.6612 305.3186 <.0001 403.0265 ## tau^2 R^2 ## Full 0.2108 ## Reduced 0.0000 NA%

Forrest Plot forest(model2, digits = 3, order = "obs") text(-3.9, 37, "Author(s) and Year", pos=4) text(5.5, 37, "Std. Dev and 95% C.I.", pos=2)

Bias Check - Fail-Safe N fsn(yi=es, vi=var, data=Afinal, type ="Rosenthal")

## ## Fail-safe N Calculation Using the Rosenthal Approach ## ## Observed Significance Level: <.0001 ## Target Significance Level: 0.05 ## ## Fail-safe N: 6421 fsn(yi=es, vi=var, data=Afinal, type ="Rosenberg")

## ## Fail-safe N Calculation Using the Rosenberg Approach ## ## Average Effect Size: 0.7215 ## Observed Significance Level: <.0001 ## Target Significance Level: 0.05

227

## ## Fail-safe N: 6211 fsn(yi=es, vi=var, data=Afinal, type ="Orwin", target=)

## ## Fail-safe N Calculation Using the Orwin Approach ## ## Average Effect Size: 0.6433 ## Target Effect Size: 0.3217 ## ## Fail-safe N: 34

Bias Check - Trim-and-Fill Plot library(metafor)

### carry out trim-and-fill analysis taf <- trimfill(model2)

### draw funnel plot with missing studies filled in funnel(taf)

Sub-Group Analyses Sub-Group Mentoring/Coaching modelmen.1<-rma(measure="SMD", yi=es, vi=var, weights=wt, method="REML", subs et=(mentor_1=="0"), data=Afinal, slab=paste(id)) modelmen.2<-rma(measure="SMD", yi=es, vi=var, weights=wt, method="REML", subs et=(mentor_1=="1"), data=Afinal, slab=paste(id)) modelmen.3<-rma(measure="SMD", yi=es, vi=var, weights=wt, mods=~mentor_1, met hod="REML", data=Afinal)

summary(modelmen.1)

## ## Random-Effects Model (k = 29; tau^2 estimator: REML) ## ## logLik deviance AIC BIC AICc ## -22.5425 45.0850 49.0850 51.7494 49.5650 ## ## tau^2 (estimated amount of total heterogeneity): 0.2484 (SE = 0.0779) ## tau (square root of estimated tau^2 value): 0.4984 ## I^2 (total heterogeneity / total variability): 90.48% ## H^2 (total variability / sampling variability): 10.51 ## ## Test for Heterogeneity: ## Q(df = 28) = 392.9245, p-val < .0001 ##

228

## Model Results: ## ## estimate se zval pval ci.lb ci.ub ## 0.7475 0.1382 5.4107 <.0001 0.4768 1.0183 *** ## ## --- ## Signif. codes: 0 '***' 0.001 '**' 0.01 '*' 0.05 '.' 0.1 ' ' 1 summary(modelmen.2)

## ## Random-Effects Model (k = 5; tau^2 estimator: REML) ## ## logLik deviance AIC BIC AICc ## 0.9981 -1.9962 2.0038 0.7764 14.0038 ## ## tau^2 (estimated amount of total heterogeneity): 0.0062 (SE = 0.0224) ## tau (square root of estimated tau^2 value): 0.0786 ## I^2 (total heterogeneity / total variability): 18.82% ## H^2 (total variability / sampling variability): 1.23 ## ## Test for Heterogeneity: ## Q(df = 4) = 4.7590, p-val = 0.3129 ## ## Model Results: ## ## estimate se zval pval ci.lb ci.ub ## 0.5701 0.0809 7.0435 <.0001 0.4114 0.7287 *** ## ## --- ## Signif. codes: 0 '***' 0.001 '**' 0.01 '*' 0.05 '.' 0.1 ' ' 1 summary(modelmen.3)

## ## Mixed-Effects Model (k = 34; tau^2 estimator: REML) ## ## logLik deviance AIC BIC AICc ## -23.9053 47.8106 53.8106 58.2078 54.6677 ## ## tau^2 (estimated amount of residual heterogeneity): 0.2180 (SE = 0.0648) ## tau (square root of estimated tau^2 value): 0.4669 ## I^2 (residual heterogeneity / unaccounted variability): 89.27% ## H^2 (unaccounted variability / sampling variability): 9.32 ## R^2 (amount of heterogeneity accounted for): 0.00% ## ## Test for Residual Heterogeneity: ## QE(df = 32) = 397.6836, p-val < .0001 ## ## Test of Moderators (coefficient(s) 2): ## QM(df = 1) = 0.4178, p-val = 0.5180

229

## ## Model Results: ## ## estimate se zval pval ci.lb ci.ub ## intrcpt 0.7475 0.1298 5.7576 <.0001 0.4931 1.0020 *** ## mentor_1 -0.1775 0.2746 -0.6464 0.5180 -0.7157 0.3607 ## ## --- ## Signif. codes: 0 '***' 0.001 '**' 0.01 '*' 0.05 '.' 0.1 ' ' 1

Sub-Group Field Experience/Rehersal modelfe.1<-rma(measure="SMD", yi=es, vi=var, weights=wt, method="REML", subse t=(fexp_2=="0"), data=Afinal, slab=paste(id)) modelfe.2<-rma(measure="SMD", yi=es, vi=var, weights=wt, method="REML", subse t=(fexp_2=="1"), data=Afinal, slab=paste(id)) modelfe.3<-rma(measure="SMD", yi=es, vi=var, weights=wt, mods=~fexp_2, method ="REML", data=Afinal)

summary(modelfe.1)

## ## Random-Effects Model (k = 30; tau^2 estimator: REML) ## ## logLik deviance AIC BIC AICc ## -21.9897 43.9795 47.9795 50.7141 48.4410 ## ## tau^2 (estimated amount of total heterogeneity): 0.2307 (SE = 0.0717) ## tau (square root of estimated tau^2 value): 0.4803 ## I^2 (total heterogeneity / total variability): 88.77% ## H^2 (total variability / sampling variability): 8.90 ## ## Test for Heterogeneity: ## Q(df = 29) = 356.4909, p-val < .0001 ## ## Model Results: ## ## estimate se zval pval ci.lb ci.ub ## 0.6475 0.1102 5.8783 <.0001 0.4316 0.8634 *** ## ## --- ## Signif. codes: 0 '***' 0.001 '**' 0.01 '*' 0.05 '.' 0.1 ' ' 1 summary(modelfe.2)

## ## Random-Effects Model (k = 4; tau^2 estimator: REML) ## ## logLik deviance AIC BIC AICc ## -1.4144 2.8288 6.8288 5.0260 18.8288

230

## ## tau^2 (estimated amount of total heterogeneity): 0.0890 (SE = 0.0978) ## tau (square root of estimated tau^2 value): 0.2983 ## I^2 (total heterogeneity / total variability): 79.97% ## H^2 (total variability / sampling variability): 4.99 ## ## Test for Heterogeneity: ## Q(df = 3) = 21.6569, p-val < .0001 ## ## Model Results: ## ## estimate se zval pval ci.lb ci.ub ## 0.9696 0.2318 4.1822 <.0001 0.5152 1.4240 *** ## ## --- ## Signif. codes: 0 '***' 0.001 '**' 0.01 '*' 0.05 '.' 0.1 ' ' 1 summary(modelfe.3)

## ## Mixed-Effects Model (k = 34; tau^2 estimator: REML) ## ## logLik deviance AIC BIC AICc ## -23.5439 47.0879 53.0879 57.4851 53.9450 ## ## tau^2 (estimated amount of residual heterogeneity): 0.2157 (SE = 0.0642) ## tau (square root of estimated tau^2 value): 0.4644 ## I^2 (residual heterogeneity / unaccounted variability): 88.37% ## H^2 (unaccounted variability / sampling variability): 8.60 ## R^2 (amount of heterogeneity accounted for): 0.00% ## ## Test for Residual Heterogeneity: ## QE(df = 32) = 378.1478, p-val < .0001 ## ## Test of Moderators (coefficient(s) 2): ## QM(df = 1) = 0.7568, p-val = 0.3843 ## ## Model Results: ## ## estimate se zval pval ci.lb ci.ub ## intrcpt 0.6475 0.1068 6.0634 <.0001 0.4382 0.8568 *** ## fexp_2 0.3221 0.3702 0.8699 0.3843 -0.4036 1.0477 ## ## --- ## Signif. codes: 0 '***' 0.001 '**' 0.01 '*' 0.05 '.' 0.1 ' ' 1

Sub-Group Goal-Setting modelgs.1<-rma(measure="SMD", yi=es, vi=var, weights=wt, method="REML", subse t=(goals_3=="0"), data=Afinal, slab=paste(id)) modelgs.2<-rma(measure="SMD", yi=es, vi=var, weights=wt, method="REML", subse

231

t=(goals_3=="1"), data=Afinal, slab=paste(id)) modelgs.3<-rma(measure="SMD", yi=es, vi=var, weights=wt, mods=~goals_3, metho d="REML", data=Afinal)

summary(modelgs.1)

## ## Random-Effects Model (k = 33; tau^2 estimator: REML) ## ## logLik deviance AIC BIC AICc ## -23.6808 47.3616 51.3616 54.2931 51.7754 ## ## tau^2 (estimated amount of total heterogeneity): 0.2167 (SE = 0.0642) ## tau (square root of estimated tau^2 value): 0.4655 ## I^2 (total heterogeneity / total variability): 89.50% ## H^2 (total variability / sampling variability): 9.53 ## ## Test for Heterogeneity: ## Q(df = 32) = 402.8298, p-val < .0001 ## ## Model Results: ## ## estimate se zval pval ci.lb ci.ub ## 0.7202 0.1172 6.1453 <.0001 0.4905 0.9499 *** ## ## --- ## Signif. codes: 0 '***' 0.001 '**' 0.01 '*' 0.05 '.' 0.1 ' ' 1 summary(modelgs.2)

## ## Fixed-Effects Model (k = 1) ## ## logLik deviance AIC BIC AICc ## 0.4561 0.0000 1.0878 -0.9122 5.0878 ## ## Test for Heterogeneity: ## Q(df = 0) = 0.0000, p-val = 1.0000 ## ## Model Results: ## ## estimate se zval pval ci.lb ci.ub ## 0.8330 0.2528 3.2947 0.0010 0.3375 1.3286 *** ## ## --- ## Signif. codes: 0 '***' 0.001 '**' 0.01 '*' 0.05 '.' 0.1 ' ' 1 summary(modelgs.3)

232

## ## Mixed-Effects Model (k = 34; tau^2 estimator: REML) ## ## logLik deviance AIC BIC AICc ## -23.6808 47.3616 53.3616 57.7588 54.2188 ## ## tau^2 (estimated amount of residual heterogeneity): 0.2167 (SE = 0.0642) ## tau (square root of estimated tau^2 value): 0.4655 ## I^2 (residual heterogeneity / unaccounted variability): 89.50% ## H^2 (unaccounted variability / sampling variability): 9.53 ## R^2 (amount of heterogeneity accounted for): 0.00% ## ## Test for Residual Heterogeneity: ## QE(df = 32) = 402.8298, p-val < .0001 ## ## Test of Moderators (coefficient(s) 2): ## QM(df = 1) = 0.0432, p-val = 0.8353 ## ## Model Results: ## ## estimate se zval pval ci.lb ci.ub ## intrcpt 0.7202 0.1172 6.1453 <.0001 0.4905 0.9499 *** ## goals_3 0.1128 0.5425 0.2079 0.8353 -0.9505 1.1761 ## ## --- ## Signif. codes: 0 '***' 0.001 '**' 0.01 '*' 0.05 '.' 0.1 ' ' 1

Sub-Group Observation modelobs.1<-rma(measure="SMD", yi=es, vi=var, weights=wt, method="REML", subs et=(observ_4=="0"), data=Afinal, slab=paste(id)) modelobs.2<-rma(measure="SMD", yi=es, vi=var, weights=wt, method="REML", subs et=(observ_4=="1"), data=Afinal, slab=paste(id)) modelobs.3<-rma(measure="SMD", yi=es, vi=var, weights=wt, mods=~observ_4, met hod="REML", data=Afinal)

summary(modelobs.1)

## ## Random-Effects Model (k = 30; tau^2 estimator: REML) ## ## logLik deviance AIC BIC AICc ## -22.5847 45.1694 49.1694 51.9040 49.6309 ## ## tau^2 (estimated amount of total heterogeneity): 0.2369 (SE = 0.0730) ## tau (square root of estimated tau^2 value): 0.4868 ## I^2 (total heterogeneity / total variability): 90.41% ## H^2 (total variability / sampling variability): 10.43 ##

233

## Test for Heterogeneity: ## Q(df = 29) = 399.5450, p-val < .0001 ## ## Model Results: ## ## estimate se zval pval ci.lb ci.ub ## 0.7151 0.1297 5.5149 <.0001 0.4609 0.9692 *** ## ## --- ## Signif. codes: 0 '***' 0.001 '**' 0.01 '*' 0.05 '.' 0.1 ' ' 1 summary(modelobs.2)

## ## Random-Effects Model (k = 4; tau^2 estimator: REML) ## ## logLik deviance AIC BIC AICc ## 0.5640 -1.1279 2.8721 1.0693 14.8721 ## ## tau^2 (estimated amount of total heterogeneity): 0.0000 (SE = 0.0276) ## tau (square root of estimated tau^2 value): 0.0016 ## I^2 (total heterogeneity / total variability): 0.01% ## H^2 (total variability / sampling variability): 1.00 ## ## Test for Heterogeneity: ## Q(df = 3) = 2.8828, p-val = 0.4101 ## ## Model Results: ## ## estimate se zval pval ci.lb ci.ub ## 0.7898 0.0924 8.5513 <.0001 0.6088 0.9708 *** ## ## --- ## Signif. codes: 0 '***' 0.001 '**' 0.01 '*' 0.05 '.' 0.1 ' ' 1 summary(modelobs.3)

## ## Mixed-Effects Model (k = 34; tau^2 estimator: REML) ## ## logLik deviance AIC BIC AICc ## -23.7224 47.4448 53.4448 57.8420 54.3019 ## ## tau^2 (estimated amount of residual heterogeneity): 0.2170 (SE = 0.0645) ## tau (square root of estimated tau^2 value): 0.4658 ## I^2 (residual heterogeneity / unaccounted variability): 89.33% ## H^2 (unaccounted variability / sampling variability): 9.37 ## R^2 (amount of heterogeneity accounted for): 0.00% ## ## Test for Residual Heterogeneity: ## QE(df = 32) = 402.4278, p-val < .0001

234

## ## Test of Moderators (coefficient(s) 2): ## QM(df = 1) = 0.0607, p-val = 0.8054 ## ## Model Results: ## ## estimate se zval pval ci.lb ci.ub ## intrcpt 0.7151 0.1244 5.7499 <.0001 0.4713 0.9588 *** ## observ_4 0.0748 0.3035 0.2464 0.8054 -0.5200 0.6695 ## ## --- ## Signif. codes: 0 '***' 0.001 '**' 0.01 '*' 0.05 '.' 0.1 ' ' 1

Sub-Group Reflection/Self-Evaluation modelref.1<-rma(measure="SMD", yi=es, vi=var, weights=wt, method="REML", subs et=(reflect_5=="0"), data=Afinal, slab=paste(id)) modelref.2<-rma(measure="SMD", yi=es, vi=var, weights=wt, method="REML", subs et=(reflect_5=="1"), data=Afinal, slab=paste(id)) modelref.3<-rma(measure="SMD", yi=es, vi=var, weights=wt, mods=~reflect_5, me thod="REML", data=Afinal)

summary(modelref.1)

## ## Random-Effects Model (k = 25; tau^2 estimator: REML) ## ## logLik deviance AIC BIC AICc ## -19.2990 38.5981 42.5981 44.9542 43.1695 ## ## tau^2 (estimated amount of total heterogeneity): 0.2331 (SE = 0.0787) ## tau (square root of estimated tau^2 value): 0.4828 ## I^2 (total heterogeneity / total variability): 90.24% ## H^2 (total variability / sampling variability): 10.25 ## ## Test for Heterogeneity: ## Q(df = 24) = 343.5518, p-val < .0001 ## ## Model Results: ## ## estimate se zval pval ci.lb ci.ub ## 0.7817 0.1430 5.4657 <.0001 0.5014 1.0621 *** ## ## --- ## Signif. codes: 0 '***' 0.001 '**' 0.01 '*' 0.05 '.' 0.1 ' ' 1 summary(modelref.2)

## ## Random-Effects Model (k = 9; tau^2 estimator: REML)

235

## ## logLik deviance AIC BIC AICc ## -6.2392 12.4785 16.4785 16.6374 18.8785 ## ## tau^2 (estimated amount of total heterogeneity): 0.1598 (SE = 0.1023) ## tau (square root of estimated tau^2 value): 0.3998 ## I^2 (total heterogeneity / total variability): 83.91% ## H^2 (total variability / sampling variability): 6.21 ## ## Test for Heterogeneity: ## Q(df = 8) = 43.3424, p-val < .0001 ## ## Model Results: ## ## estimate se zval pval ci.lb ci.ub ## 0.5239 0.1768 2.9635 0.0030 0.1774 0.8704 ** ## ## --- ## Signif. codes: 0 '***' 0.001 '**' 0.01 '*' 0.05 '.' 0.1 ' ' 1 summary(modelref.3)

## ## Mixed-Effects Model (k = 34; tau^2 estimator: REML) ## ## logLik deviance AIC BIC AICc ## -25.5933 51.1866 57.1866 61.5838 58.0438 ## ## tau^2 (estimated amount of residual heterogeneity): 0.2171 (SE = 0.0645) ## tau (square root of estimated tau^2 value): 0.4660 ## I^2 (residual heterogeneity / unaccounted variability): 89.17% ## H^2 (unaccounted variability / sampling variability): 9.23 ## R^2 (amount of heterogeneity accounted for): 0.00% ## ## Test for Residual Heterogeneity: ## QE(df = 32) = 386.8943, p-val < .0001 ## ## Test of Moderators (coefficient(s) 2): ## QM(df = 1) = 1.0998, p-val = 0.2943 ## ## Model Results: ## ## estimate se zval pval ci.lb ci.ub ## intrcpt 0.7817 0.1383 5.6534 <.0001 0.5107 1.0528 *** ## reflect_5 -0.2578 0.2459 -1.0487 0.2943 -0.7397 0.2240 ## ## --- ## Signif. codes: 0 '***' 0.001 '**' 0.01 '*' 0.05 '.' 0.1 ' ' 1

Sub-Group Hands-On Learning

236

modelhol.1<-rma(measure="SMD", yi=es, vi=var, weights=wt, method="REML", subs et=(hands_6=="0"), data=Afinal, slab=paste(id)) modelhol.2<-rma(measure="SMD", yi=es, vi=var, weights=wt, method="REML", subs et=(hands_6=="1"), data=Afinal, slab=paste(id)) modelhol.3<-rma(measure="SMD", yi=es, vi=var, weights=wt, mods=~hands_6, meth od="REML", data=Afinal)

summary(modelhol.1)

## ## Random-Effects Model (k = 21; tau^2 estimator: REML) ## ## logLik deviance AIC BIC AICc ## -13.3706 26.7412 30.7412 32.7327 31.4471 ## ## tau^2 (estimated amount of total heterogeneity): 0.1806 (SE = 0.0699) ## tau (square root of estimated tau^2 value): 0.4250 ## I^2 (total heterogeneity / total variability): 87.59% ## H^2 (total variability / sampling variability): 8.05 ## ## Test for Heterogeneity: ## Q(df = 20) = 199.3736, p-val < .0001 ## ## Model Results: ## ## estimate se zval pval ci.lb ci.ub ## 0.6408 0.1432 4.4752 <.0001 0.3601 0.9214 *** ## ## --- ## Signif. codes: 0 '***' 0.001 '**' 0.01 '*' 0.05 '.' 0.1 ' ' 1 summary(modelhol.2)

## ## Random-Effects Model (k = 13; tau^2 estimator: REML) ## ## logLik deviance AIC BIC AICc ## -10.5252 21.0503 25.0503 26.0201 26.3837 ## ## tau^2 (estimated amount of total heterogeneity): 0.2701 (SE = 0.1273) ## tau (square root of estimated tau^2 value): 0.5197 ## I^2 (total heterogeneity / total variability): 90.57% ## H^2 (total variability / sampling variability): 10.60 ## ## Test for Heterogeneity: ## Q(df = 12) = 187.5567, p-val < .0001 ## ## Model Results: ##

237

## estimate se zval pval ci.lb ci.ub ## 0.8686 0.1796 4.8369 <.0001 0.5166 1.2206 *** ## ## --- ## Signif. codes: 0 '***' 0.001 '**' 0.01 '*' 0.05 '.' 0.1 ' ' 1 summary(modelhol.3)

## ## Mixed-Effects Model (k = 34; tau^2 estimator: REML) ## ## logLik deviance AIC BIC AICc ## -24.2469 48.4937 54.4937 58.8909 55.3509 ## ## tau^2 (estimated amount of residual heterogeneity): 0.2164 (SE = 0.0644) ## tau (square root of estimated tau^2 value): 0.4652 ## I^2 (residual heterogeneity / unaccounted variability): 89.09% ## H^2 (unaccounted variability / sampling variability): 9.17 ## R^2 (amount of heterogeneity accounted for): 0.00% ## ## Test for Residual Heterogeneity: ## QE(df = 32) = 386.9302, p-val < .0001 ## ## Test of Moderators (coefficient(s) 2): ## QM(df = 1) = 1.0259, p-val = 0.3111 ## ## Model Results: ## ## estimate se zval pval ci.lb ci.ub ## intrcpt 0.6408 0.1560 4.1069 <.0001 0.3350 0.9466 *** ## hands_6 0.2278 0.2249 1.0129 0.3111 -0.2130 0.6687 ## ## --- ## Signif. codes: 0 '***' 0.001 '**' 0.01 '*' 0.05 '.' 0.1 ' ' 1

Sub-Group Work Sample Analysis modelws.1<-rma(measure="SMD", yi=es, vi=var, weights=wt, method="REML", subse t=(wsan_7=="0"), data=Afinal, slab=paste(id)) modelws.2<-rma(measure="SMD", yi=es, vi=var, weights=wt, method="REML", subse t=(wsan_7=="1"), data=Afinal, slab=paste(id)) modelws.3<-rma(measure="SMD", yi=es, vi=var, weights=wt, mods=~wsan_7, method ="REML", data=Afinal)

summary(modelws.1)

## ## Random-Effects Model (k = 27; tau^2 estimator: REML) ## ## logLik deviance AIC BIC AICc

238

## -20.5624 41.1248 45.1248 47.6410 45.6465 ## ## tau^2 (estimated amount of total heterogeneity): 0.2388 (SE = 0.0789) ## tau (square root of estimated tau^2 value): 0.4887 ## I^2 (total heterogeneity / total variability): 89.57% ## H^2 (total variability / sampling variability): 9.58 ## ## Test for Heterogeneity: ## Q(df = 26) = 356.5687, p-val < .0001 ## ## Model Results: ## ## estimate se zval pval ci.lb ci.ub ## 0.7250 0.1468 4.9386 <.0001 0.4372 1.0127 *** ## ## --- ## Signif. codes: 0 '***' 0.001 '**' 0.01 '*' 0.05 '.' 0.1 ' ' 1 summary(modelws.2)

## ## Random-Effects Model (k = 7; tau^2 estimator: REML) ## ## logLik deviance AIC BIC AICc ## -2.9077 5.8154 9.8154 9.3989 13.8154 ## ## tau^2 (estimated amount of total heterogeneity): 0.1317 (SE = 0.0908) ## tau (square root of estimated tau^2 value): 0.3629 ## I^2 (total heterogeneity / total variability): 85.98% ## H^2 (total variability / sampling variability): 7.13 ## ## Test for Heterogeneity: ## Q(df = 6) = 46.4083, p-val < .0001 ## ## Model Results: ## ## estimate se zval pval ci.lb ci.ub ## 0.7109 0.1554 4.5743 <.0001 0.4063 1.0155 *** ## ## --- ## Signif. codes: 0 '***' 0.001 '**' 0.01 '*' 0.05 '.' 0.1 ' ' 1 summary(modelws.3)

## ## Mixed-Effects Model (k = 34; tau^2 estimator: REML) ## ## logLik deviance AIC BIC AICc ## -23.8063 47.6125 53.6125 58.0097 54.4697 ## ## tau^2 (estimated amount of residual heterogeneity): 0.2173 (SE = 0.0647)

239

## tau (square root of estimated tau^2 value): 0.4661 ## I^2 (residual heterogeneity / unaccounted variability): 89.18% ## H^2 (unaccounted variability / sampling variability): 9.24 ## R^2 (amount of heterogeneity accounted for): 0.00% ## ## Test for Residual Heterogeneity: ## QE(df = 32) = 402.9770, p-val < .0001 ## ## Test of Moderators (coefficient(s) 2): ## QM(df = 1) = 0.0034, p-val = 0.9533 ## ## Model Results: ## ## estimate se zval pval ci.lb ci.ub ## intrcpt 0.7250 0.1403 5.1661 <.0001 0.4499 1.0000 *** ## wsan_7 -0.0140 0.2400 -0.0585 0.9533 -0.4844 0.4563 ## ## --- ## Signif. codes: 0 '***' 0.001 '**' 0.01 '*' 0.05 '.' 0.1 ' ' 1

Sub-Group Lesson Planning modellp.1<-rma(measure="SMD", yi=es, vi=var, weights=wt, method="REML", subse t=(lps_8=="0"), data=Afinal, slab=paste(id)) modellp.2<-rma(measure="SMD", yi=es, vi=var, weights=wt, method="REML", subse t=(lps_8=="1"), data=Afinal, slab=paste(id)) modellp.3<-rma(measure="SMD", yi=es, vi=var, weights=wt, mods=~lps_8, method= "REML", data=Afinal)

summary(modellp.1)

## ## Random-Effects Model (k = 21; tau^2 estimator: REML) ## ## logLik deviance AIC BIC AICc ## -15.2133 30.4267 34.4267 36.4181 35.1325 ## ## tau^2 (estimated amount of total heterogeneity): 0.2379 (SE = 0.0884) ## tau (square root of estimated tau^2 value): 0.4878 ## I^2 (total heterogeneity / total variability): 88.84% ## H^2 (total variability / sampling variability): 8.96 ## ## Test for Heterogeneity: ## Q(df = 20) = 294.3532, p-val < .0001 ## ## Model Results: ## ## estimate se zval pval ci.lb ci.ub ## 0.6003 0.1343 4.4691 <.0001 0.3370 0.8636 ***

240

## ## --- ## Signif. codes: 0 '***' 0.001 '**' 0.01 '*' 0.05 '.' 0.1 ' ' 1 summary(modellp.2)

## ## Random-Effects Model (k = 13; tau^2 estimator: REML) ## ## logLik deviance AIC BIC AICc ## -6.3864 12.7727 16.7727 17.7426 18.1061 ## ## tau^2 (estimated amount of total heterogeneity): 0.1295 (SE = 0.0683) ## tau (square root of estimated tau^2 value): 0.3599 ## I^2 (total heterogeneity / total variability): 84.95% ## H^2 (total variability / sampling variability): 6.64 ## ## Test for Heterogeneity: ## Q(df = 12) = 86.1150, p-val < .0001 ## ## Model Results: ## ## estimate se zval pval ci.lb ci.ub ## 0.8588 0.1566 5.4831 <.0001 0.5518 1.1658 *** ## ## --- ## Signif. codes: 0 '***' 0.001 '**' 0.01 '*' 0.05 '.' 0.1 ' ' 1 summary(modellp.3)

## ## Mixed-Effects Model (k = 34; tau^2 estimator: REML) ## ## logLik deviance AIC BIC AICc ## -22.0709 44.1417 50.1417 54.5390 50.9989 ## ## tau^2 (estimated amount of residual heterogeneity): 0.2002 (SE = 0.0602) ## tau (square root of estimated tau^2 value): 0.4474 ## I^2 (residual heterogeneity / unaccounted variability): 88.18% ## H^2 (unaccounted variability / sampling variability): 8.46 ## R^2 (amount of heterogeneity accounted for): 5.04% ## ## Test for Residual Heterogeneity: ## QE(df = 32) = 380.4683, p-val < .0001 ## ## Test of Moderators (coefficient(s) 2): ## QM(df = 1) = 1.2740, p-val = 0.2590 ## ## Model Results: ## ## estimate se zval pval ci.lb ci.ub

241

## intrcpt 0.6003 0.1241 4.8368 <.0001 0.3570 0.8436 *** ## lps_8 0.2585 0.2290 1.1287 0.2590 -0.1904 0.7074 ## ## --- ## Signif. codes: 0 '***' 0.001 '**' 0.01 '*' 0.05 '.' 0.1 ' ' 1

Sub-Group Study quality modqs.1<-rma(yi=es, vi=var, weights=wt, method="REML", subset=(qs_rank=="low" ), data=Afinal, slab=paste(id)) modqs.2<-rma(yi=es, vi=var, weights=wt, method="REML", subset=(qs_rank=="ave" ), data=Afinal, slab=paste(id)) modqs.3<-rma(yi=es, vi=var, weights=wt, method="REML", subset=(qs_rank=="high "), data=Afinal, slab=paste(id)) modqs.4<-rma(yi=es, vi=var, weights=wt, mods=~qs_rank, method="REML", data=Af inal)

summary(modqs.1)

## ## Random-Effects Model (k = 5; tau^2 estimator: REML) ## ## logLik deviance AIC BIC AICc ## -1.6033 3.2066 7.2066 5.9792 19.2066 ## ## tau^2 (estimated amount of total heterogeneity): 0.0995 (SE = 0.1017) ## tau (square root of estimated tau^2 value): 0.3154 ## I^2 (total heterogeneity / total variability): 73.06% ## H^2 (total variability / sampling variability): 3.71 ## ## Test for Heterogeneity: ## Q(df = 4) = 19.7899, p-val = 0.0005 ## ## Model Results: ## ## estimate se zval pval ci.lb ci.ub ## 0.1214 0.1879 0.6463 0.5181 -0.2468 0.4896 ## ## --- ## Signif. codes: 0 '***' 0.001 '**' 0.01 '*' 0.05 '.' 0.1 ' ' 1 summary(modqs.2)

## ## Random-Effects Model (k = 20; tau^2 estimator: REML) ## ## logLik deviance AIC BIC AICc ## -14.2991 28.5983 32.5983 34.4872 33.3483 ## ## tau^2 (estimated amount of total heterogeneity): 0.2156 (SE = 0.0826)

242

## tau (square root of estimated tau^2 value): 0.4643 ## I^2 (total heterogeneity / total variability): 90.10% ## H^2 (total variability / sampling variability): 10.11 ## ## Test for Heterogeneity: ## Q(df = 19) = 283.0747, p-val < .0001 ## ## Model Results: ## ## estimate se zval pval ci.lb ci.ub ## 0.7641 0.1561 4.8951 <.0001 0.4581 1.0700 *** ## ## --- ## Signif. codes: 0 '***' 0.001 '**' 0.01 '*' 0.05 '.' 0.1 ' ' 1 summary(modqs.3)

## ## Random-Effects Model (k = 9; tau^2 estimator: REML) ## ## logLik deviance AIC BIC AICc ## -3.3080 6.6161 10.6161 10.7750 13.0161 ## ## tau^2 (estimated amount of total heterogeneity): 0.0916 (SE = 0.0654) ## tau (square root of estimated tau^2 value): 0.3026 ## I^2 (total heterogeneity / total variability): 74.62% ## H^2 (total variability / sampling variability): 3.94 ## ## Test for Heterogeneity: ## Q(df = 8) = 33.6982, p-val < .0001 ## ## Model Results: ## ## estimate se zval pval ci.lb ci.ub ## 0.8995 0.1307 6.8813 <.0001 0.6433 1.1557 *** ## ## --- ## Signif. codes: 0 '***' 0.001 '**' 0.01 '*' 0.05 '.' 0.1 ' ' 1 summary(modqs.4)

## ## Mixed-Effects Model (k = 34; tau^2 estimator: REML) ## ## logLik deviance AIC BIC AICc ## -20.1414 40.2827 48.2827 54.0187 49.8212 ## ## tau^2 (estimated amount of residual heterogeneity): 0.1718 (SE = 0.0538) ## tau (square root of estimated tau^2 value): 0.4145 ## I^2 (residual heterogeneity / unaccounted variability): 86.62% ## H^2 (unaccounted variability / sampling variability): 7.47

243

## R^2 (amount of heterogeneity accounted for): 18.52% ## ## Test for Residual Heterogeneity: ## QE(df = 31) = 336.5628, p-val < .0001 ## ## Test of Moderators (coefficient(s) 2:3): ## QM(df = 2) = 7.5299, p-val = 0.0232 ## ## Model Results: ## ## estimate se zval pval ci.lb ci.ub ## intrcpt 0.7641 0.1401 5.4520 <.0001 0.4894 1.0387 *** ## qs_rankhigh 0.1355 0.2209 0.6133 0.5397 -0.2975 0.5684 ## qs_ranklow -0.6426 0.2754 -2.3335 0.0196 -1.1824 -0.1029 * ## ## --- ## Signif. codes: 0 '***' 0.001 '**' 0.01 '*' 0.05 '.' 0.1 ' ' 1

Sub-Group Validity modelv.1<-rma(yi=es, vi=var, weights=wt, method="REML", subset=(valid=="SST") , data=Afinal, slab=paste(id)) modelv.2<-rma(yi=es, vi=var, weights=wt, method="REML", subset=(valid=="PVM") , data=Afinal, slab=paste(id)) modelv.3<-rma(yi=es, vi=var, weights=wt, method="REML", subset=(valid=="PVWOM "), data=Afinal, slab=paste(id)) modelv.4<-rma(yi=es, vi=var, weights=wt, mods=~valid, method="REML", data=Afi nal)

summary(modelv.1)

## ## Random-Effects Model (k = 4; tau^2 estimator: REML) ## ## logLik deviance AIC BIC AICc ## -3.4053 6.8106 10.8106 9.0078 22.8106 ## ## tau^2 (estimated amount of total heterogeneity): 0.5508 (SE = 0.4651) ## tau (square root of estimated tau^2 value): 0.7422 ## I^2 (total heterogeneity / total variability): 97.08% ## H^2 (total variability / sampling variability): 34.27 ## ## Test for Heterogeneity: ## Q(df = 3) = 140.6186, p-val < .0001 ## ## Model Results: ## ## estimate se zval pval ci.lb ci.ub ## 1.0470 0.4029 2.5986 0.0094 0.2573 1.8366 **

244

## ## --- ## Signif. codes: 0 '***' 0.001 '**' 0.01 '*' 0.05 '.' 0.1 ' ' 1 summary(modelv.2)

## ## Random-Effects Model (k = 13; tau^2 estimator: REML) ## ## logLik deviance AIC BIC AICc ## -5.6237 11.2474 15.2474 16.2172 16.5807 ## ## tau^2 (estimated amount of total heterogeneity): 0.1229 (SE = 0.0617) ## tau (square root of estimated tau^2 value): 0.3506 ## I^2 (total heterogeneity / total variability): 84.01% ## H^2 (total variability / sampling variability): 6.25 ## ## Test for Heterogeneity: ## Q(df = 12) = 80.2951, p-val < .0001 ## ## Model Results: ## ## estimate se zval pval ci.lb ci.ub ## 0.5422 0.1130 4.7976 <.0001 0.3207 0.7638 *** ## ## --- ## Signif. codes: 0 '***' 0.001 '**' 0.01 '*' 0.05 '.' 0.1 ' ' 1 summary(modelv.3)

## ## Random-Effects Model (k = 17; tau^2 estimator: REML) ## ## logLik deviance AIC BIC AICc ## -11.6220 23.2440 27.2440 28.7892 28.1671 ## ## tau^2 (estimated amount of total heterogeneity): 0.1868 (SE = 0.0863) ## tau (square root of estimated tau^2 value): 0.4322 ## I^2 (total heterogeneity / total variability): 83.21% ## H^2 (total variability / sampling variability): 5.96 ## ## Test for Heterogeneity: ## Q(df = 16) = 136.4507, p-val < .0001 ## ## Model Results: ## ## estimate se zval pval ci.lb ci.ub ## 0.7554 0.2029 3.7224 0.0002 0.3577 1.1531 *** ## ## --- ## Signif. codes: 0 '***' 0.001 '**' 0.01 '*' 0.05 '.' 0.1 ' ' 1

245

summary(modelv.4)

## ## Mixed-Effects Model (k = 34; tau^2 estimator: REML) ## ## logLik deviance AIC BIC AICc ## -22.1226 44.2453 52.2453 57.9812 53.7837 ## ## tau^2 (estimated amount of residual heterogeneity): 0.2028 (SE = 0.0621) ## tau (square root of estimated tau^2 value): 0.4504 ## I^2 (residual heterogeneity / unaccounted variability): 87.98% ## H^2 (unaccounted variability / sampling variability): 8.32 ## R^2 (amount of heterogeneity accounted for): 3.78% ## ## Test for Residual Heterogeneity: ## QE(df = 31) = 357.3643, p-val < .0001 ## ## Test of Moderators (coefficient(s) 2:3): ## QM(df = 2) = 3.2375, p-val = 0.1981 ## ## Model Results: ## ## estimate se zval pval ci.lb ci.ub ## intrcpt 0.5422 0.1412 3.8408 0.0001 0.2655 0.8189 *** ## validPVWOM 0.2132 0.2539 0.8394 0.4012 -0.2846 0.7109 ## validSST 0.5047 0.2867 1.7605 0.0783 -0.0572 1.0666 . ## ## --- ## Signif. codes: 0 '***' 0.001 '**' 0.01 '*' 0.05 '.' 0.1 ' ' 1

Sub-Group Reliability modelrel.1<-rma(measure="SMD", yi=es, vi=var, weights=wt, method="REML", subs et=(reliable=="0"), data=Afinal, slab=paste(id)) modelrel.2<-rma(measure="SMD", yi=es, vi=var, weights=wt, method="REML", subs et=(reliable=="1"), data=Afinal, slab=paste(id)) modelrel.3<-rma(measure="SMD", yi=es, vi=var, weights=wt, mods=~reliable, met hod="REML", data=Afinal)

summary(modelrel.1)

## ## Random-Effects Model (k = 14; tau^2 estimator: REML) ## ## logLik deviance AIC BIC AICc ## -8.2510 16.5020 20.5020 21.6319 21.7020 ## ## tau^2 (estimated amount of total heterogeneity): 0.1501 (SE = 0.0766) ## tau (square root of estimated tau^2 value): 0.3875 ## I^2 (total heterogeneity / total variability): 84.51%

246

## H^2 (total variability / sampling variability): 6.45 ## ## Test for Heterogeneity: ## Q(df = 13) = 93.4380, p-val < .0001 ## ## Model Results: ## ## estimate se zval pval ci.lb ci.ub ## 0.8241 0.1768 4.6615 <.0001 0.4776 1.1706 *** ## ## --- ## Signif. codes: 0 '***' 0.001 '**' 0.01 '*' 0.05 '.' 0.1 ' ' 1 summary(modelrel.2)

## ## Random-Effects Model (k = 20; tau^2 estimator: REML) ## ## logLik deviance AIC BIC AICc ## -14.5401 29.0801 33.0801 34.9690 33.8301 ## ## tau^2 (estimated amount of total heterogeneity): 0.2426 (SE = 0.0909) ## tau (square root of estimated tau^2 value): 0.4925 ## I^2 (total heterogeneity / total variability): 90.13% ## H^2 (total variability / sampling variability): 10.13 ## ## Test for Heterogeneity: ## Q(df = 19) = 298.6615, p-val < .0001 ## ## Model Results: ## ## estimate se zval pval ci.lb ci.ub ## 0.6429 0.1330 4.8331 <.0001 0.3822 0.9036 *** ## ## --- ## Signif. codes: 0 '***' 0.001 '**' 0.01 '*' 0.05 '.' 0.1 ' ' 1 summary(modelrel.3)

## ## Mixed-Effects Model (k = 34; tau^2 estimator: REML) ## ## logLik deviance AIC BIC AICc ## -23.0683 46.1367 52.1367 56.5339 52.9938 ## ## tau^2 (estimated amount of residual heterogeneity): 0.2103 (SE = 0.0628) ## tau (square root of estimated tau^2 value): 0.4586 ## I^2 (residual heterogeneity / unaccounted variability): 88.64% ## H^2 (unaccounted variability / sampling variability): 8.81 ## R^2 (amount of heterogeneity accounted for): 0.25% ##

247

## Test for Residual Heterogeneity: ## QE(df = 32) = 392.0995, p-val < .0001 ## ## Test of Moderators (coefficient(s) 2): ## QM(df = 1) = 0.5600, p-val = 0.4542 ## ## Model Results: ## ## estimate se zval pval ci.lb ci.ub ## intrcpt 0.8241 0.2076 3.9697 <.0001 0.4172 1.2310 *** ## reliable -0.1812 0.2421 -0.7484 0.4542 -0.6557 0.2933 ## ## --- ## Signif. codes: 0 '***' 0.001 '**' 0.01 '*' 0.05 '.' 0.1 ' ' 1

248

APPENDIX L R CODE MARKDOWN DOCUMENT - BELIEFS

Data Summary, Main Effect Analyses, and Model Fit Comparison

#read data Bfinal <- read.csv("Agg_B_ES_Full.csv") summary(Bfinal)

## Study id n ## Alayyar et al. (2012) :1 AJ73 :1 Min. : 14.00 ## Anderson & Maninger (2007):1 BW93 :1 1st Qu.: 40.50 ## Bai and Ertmer (2008) :1 CM33 :1 Median : 73.50 ## Cengiz (2015) :1 DG28 :1 Mean : 91.17 ## Clark et al. (2015) :1 EY32 :1 3rd Qu.: 80.25 ## Jang (2008) :1 EZ23 :1 Max. :424.00 ## (Other) :6 (Other):6 ## es var wt mentor_1 ## Min. :-0.5922 Min. :0.004024 Min. : 9.496 Min. :0.00000 ## 1st Qu.:-0.0471 1st Qu.:0.022847 1st Qu.: 20.559 1st Qu.:0.00000 ## Median : 0.2285 Median :0.030465 Median : 32.838 Median :0.00000 ## Mean : 0.2824 Mean :0.039762 Mean : 49.519 Mean :0.08333 ## 3rd Qu.: 0.6041 3rd Qu.:0.048641 3rd Qu.: 43.776 3rd Qu.:0.00000 ## Max. : 1.1437 Max. :0.105306 Max. :248.487 Max. :1.00000 ## ## fexp_2 goals_3 observ_4 reflect_5 ## Min. :0.0000 Min. :0 Min. :0.00000 Min. :0.00000 ## 1st Qu.:0.0000 1st Qu.:0 1st Qu.:0.00000 1st Qu.:0.00000 ## Median :0.0000 Median :0 Median :0.00000 Median :0.00000 ## Mean :0.1667 Mean :0 Mean :0.08333 Mean :0.08333 ## 3rd Qu.:0.0000 3rd Qu.:0 3rd Qu.:0.00000 3rd Qu.:0.00000 ## Max. :1.0000 Max. :0 Max. :1.00000 Max. :1.00000 ## ## hands_6 wsan_7 lps_8 q_score qs_rank ## Min. :0.00 Min. :0.0000 Min. :0.0000 Min. :10.00 ave :5 ## 1st Qu.:0.00 1st Qu.:0.0000 1st Qu.:0.0000 1st Qu.:13.50 high:4 ## Median :0.00 Median :0.0000 Median :0.0000 Median :16.00 low :3 ## Mean :0.25 Mean :0.1667 Mean :0.3333 Mean :15.83 ## 3rd Qu.:0.25 3rd Qu.:0.0000 3rd Qu.:1.0000 3rd Qu.:18.00 ## Max. :1.00 Max. :1.0000 Max. :1.0000 Max. :23.00 ## ## reliable valid self_rep ## Min. :0.0000 PVM :5 Min. :1 ## 1st Qu.:0.0000 PVWOM:4 1st Qu.:1 ## Median :0.0000 SST :3 Median :1 ## Mean :0.4167 Mean :1 ## 3rd Qu.:1.0000 3rd Qu.:1 ## Max. :1.0000 Max. :1 ##

249

#Fixed-Effect Modeling library(metafor)

## Warning: package 'metafor' was built under R version 3.4.1

## Loading required package: Matrix

## Warning: package 'Matrix' was built under R version 3.4.2

## ## Fixed-Effects Model (k = 12) ## ## logLik deviance AIC BIC AICc ## -52.9306 125.7372 107.8612 108.3461 108.2612 ## ## Test for Heterogeneity: ## Q(df = 11) = 125.7372, p-val < .0001 ## ## Model Results: ## ## estimate se zval pval ci.lb ci.ub ## 0.4741 0.0410 11.5574 <.0001 0.3937 0.5545 *** ## ## --- ## Signif. codes: 0 '***' 0.001 '**' 0.01 '*' 0.05 '.' 0.1 ' ' 1

#Random-Effects Modeling model2<-rma(measure="SMD", yi=es, vi=var, weights=wt, method="REML", data=Bfi nal,slab=paste(Study)) summary(model2)

## ## Random-Effects Model (k = 12; tau^2 estimator: REML) ## ## logLik deviance AIC BIC AICc ## -9.1250 18.2500 22.2500 23.0458 23.7500 ## ## tau^2 (estimated amount of total heterogeneity): 0.2326 (SE = 0.1147) ## tau (square root of estimated tau^2 value): 0.4823 ## I^2 (total heterogeneity / total variability): 90.82% ## H^2 (total variability / sampling variability): 10.89

250

## ## Test for Heterogeneity: ## Q(df = 11) = 125.7372, p-val < .0001 ## ## Model Results: ## ## estimate se zval pval ci.lb ci.ub ## 0.4741 0.2263 2.0948 0.0362 0.0305 0.9177 * ## ## --- ## Signif. codes: 0 '***' 0.001 '**' 0.01 '*' 0.05 '.' 0.1 ' ' 1

#Model Fit Assessment anova(model2, model1)

## df AIC BIC AICc logLik LRT pval QE ## Full 2 22.2500 23.0458 23.7500 -9.1250 125.7372 ## Reduced 1 109.9256 110.3235 110.3701 -53.9628 89.6757 <.0001 125.7372 ## tau^2 R^2 ## Full 0.2326 ## Reduced 0.0000 NA%

Forrest Plot forest(model2, digits = 3, order = "obs") text(-3.9, 37, "Author(s) and Year", pos=4) text(5.5, 37, "Std. Dev and 95% C.I.", pos=2)

Bias Check - Fail-Safe N fsn(yi=es, vi=var, data=Bfinal, type ="Rosenthal")

## ## Fail-safe N Calculation Using the Rosenthal Approach ## ## Observed Significance Level: <.0001 ## Target Significance Level: 0.05 ## ## Fail-safe N: 270 fsn(yi=es, vi=var, data=Bfinal, type ="Rosenberg")

## ## Fail-safe N Calculation Using the Rosenberg Approach ## ## Average Effect Size: 0.4741 ## Observed Significance Level: <.0001 ## Target Significance Level: 0.05 ## ## Fail-safe N: 406

251

fsn(yi=es, vi=var, data=Bfinal, type ="Orwin", target=)

## ## Fail-safe N Calculation Using the Orwin Approach ## ## Average Effect Size: 0.2824 ## Target Effect Size: 0.1412 ## ## Fail-safe N: 12

Bias Check - Trim-and-Fill Plot library(metafor)

### carry out trim-and-fill analysis taf <- trimfill(model2)

### draw funnel plot with missing studies filled in funnel(taf)

Sub-Group Analyses Sub-Group Mentoring/Coaching modelmen.1<-rma(measure="SMD", yi=es, vi=var, weights=wt, method="REML", subs et=(mentor_1=="0"), data=Bfinal, slab=paste(id)) modelmen.2<-rma(measure="SMD", yi=es, vi=var, weights=wt, method="REML", subs et=(mentor_1=="1"), data=Bfinal, slab=paste(id)) modelmen.3<-rma(measure="SMD", yi=es, vi=var, weights=wt, mods=~mentor_1, met hod="REML", data=Bfinal)

summary(modelmen.1)

## ## Random-Effects Model (k = 11; tau^2 estimator: REML) ## ## logLik deviance AIC BIC AICc ## -8.7090 17.4181 21.4181 22.0232 23.1324 ## ## tau^2 (estimated amount of total heterogeneity): 0.2526 (SE = 0.1299) ## tau (square root of estimated tau^2 value): 0.5026 ## I^2 (total heterogeneity / total variability): 91.34% ## H^2 (total variability / sampling variability): 11.55 ## ## Test for Heterogeneity: ## Q(df = 10) = 117.3389, p-val < .0001 ## ## Model Results: ## ## estimate se zval pval ci.lb ci.ub

252

## 0.5075 0.2507 2.0242 0.0430 0.0161 0.9989 * ## ## --- ## Signif. codes: 0 '***' 0.001 '**' 0.01 '*' 0.05 '.' 0.1 ' ' 1 summary(modelmen.2)

## ## Fixed-Effects Model (k = 1) ## ## logLik deviance AIC BIC AICc ## 0.9673 0.0000 0.0654 -1.9346 4.0654 ## ## Test for Heterogeneity: ## Q(df = 0) = 0.0000, p-val = 1.0000 ## ## Model Results: ## ## estimate se zval pval ci.lb ci.ub ## 0.0510 0.1516 0.3366 0.7364 -0.2462 0.3483 ## ## --- ## Signif. codes: 0 '***' 0.001 '**' 0.01 '*' 0.05 '.' 0.1 ' ' 1 summary(modelmen.3)

## ## Mixed-Effects Model (k = 12; tau^2 estimator: REML) ## ## logLik deviance AIC BIC AICc ## -8.7090 17.4181 23.4181 24.3258 27.4181 ## ## tau^2 (estimated amount of residual heterogeneity): 0.2526 (SE = 0.1299) ## tau (square root of estimated tau^2 value): 0.5026 ## I^2 (residual heterogeneity / unaccounted variability): 91.34% ## H^2 (unaccounted variability / sampling variability): 11.55 ## R^2 (amount of heterogeneity accounted for): 0.00% ## ## Test for Residual Heterogeneity: ## QE(df = 10) = 117.3389, p-val < .0001 ## ## Test of Moderators (coefficient(s) 2): ## QM(df = 1) = 0.6157, p-val = 0.4326 ## ## Model Results: ## ## estimate se zval pval ci.lb ci.ub ## intrcpt 0.5075 0.2507 2.0242 0.0430 0.0161 0.9989 * ## mentor_1 -0.4565 0.5817 -0.7847 0.4326 -1.5967 0.6837 ##

253

## --- ## Signif. codes: 0 '***' 0.001 '**' 0.01 '*' 0.05 '.' 0.1 ' ' 1

Sub-Group Field Experience/Rehersal modelfe.1<-rma(measure="SMD", yi=es, vi=var, weights=wt, method="REML", subse t=(fexp_2=="0"), data=Bfinal, slab=paste(id)) modelfe.2<-rma(measure="SMD", yi=es, vi=var, weights=wt, method="REML", subse t=(fexp_2=="1"), data=Bfinal, slab=paste(id)) modelfe.3<-rma(measure="SMD", yi=es, vi=var, weights=wt, mods=~fexp_2, method ="REML", data=Bfinal)

summary(modelfe.1)

## ## Random-Effects Model (k = 10; tau^2 estimator: REML) ## ## logLik deviance AIC BIC AICc ## -7.3380 14.6760 18.6760 19.0704 20.6760 ## ## tau^2 (estimated amount of total heterogeneity): 0.2545 (SE = 0.1398) ## tau (square root of estimated tau^2 value): 0.5044 ## I^2 (total heterogeneity / total variability): 88.18% ## H^2 (total variability / sampling variability): 8.46 ## ## Test for Heterogeneity: ## Q(df = 9) = 77.7053, p-val < .0001 ## ## Model Results: ## ## estimate se zval pval ci.lb ci.ub ## 0.2643 0.1885 1.4020 0.1609 -0.1052 0.6338 ## ## --- ## Signif. codes: 0 '***' 0.001 '**' 0.01 '*' 0.05 '.' 0.1 ' ' 1 summary(modelfe.2)

## ## Random-Effects Model (k = 2; tau^2 estimator: REML) ## ## logLik deviance AIC BIC AICc ## -1.0116 2.0231 6.0231 2.0231 18.0231 ## ## tau^2 (estimated amount of total heterogeneity): 0.2695 (SE = 0.4002) ## tau (square root of estimated tau^2 value): 0.5191 ## I^2 (total heterogeneity / total variability): 95.23% ## H^2 (total variability / sampling variability): 20.95 ## ## Test for Heterogeneity:

254

## Q(df = 1) = 20.9481, p-val < .0001 ## ## Model Results: ## ## estimate se zval pval ci.lb ci.ub ## 0.6913 0.4523 1.5284 0.1264 -0.1952 1.5779 ## ## --- ## Signif. codes: 0 '***' 0.001 '**' 0.01 '*' 0.05 '.' 0.1 ' ' 1 summary(modelfe.3)

## ## Mixed-Effects Model (k = 12; tau^2 estimator: REML) ## ## logLik deviance AIC BIC AICc ## -8.3601 16.7201 22.7201 23.6279 26.7201 ## ## tau^2 (estimated amount of residual heterogeneity): 0.2563 (SE = 0.1319) ## tau (square root of estimated tau^2 value): 0.5062 ## I^2 (residual heterogeneity / unaccounted variability): 89.65% ## H^2 (unaccounted variability / sampling variability): 9.66 ## R^2 (amount of heterogeneity accounted for): 0.00% ## ## Test for Residual Heterogeneity: ## QE(df = 10) = 98.6534, p-val < .0001 ## ## Test of Moderators (coefficient(s) 2): ## QM(df = 1) = 0.7912, p-val = 0.3737 ## ## Model Results: ## ## estimate se zval pval ci.lb ci.ub ## intrcpt 0.2643 0.1891 1.3975 0.1623 -0.1064 0.6349 ## fexp_2 0.4270 0.4801 0.8895 0.3737 -0.5139 1.3680 ## ## --- ## Signif. codes: 0 '***' 0.001 '**' 0.01 '*' 0.05 '.' 0.1 ' ' 1

Sub-Group Goal-Setting

#modelgs.1<-rma(measure="SMD", yi=es, vi=var, weights=wt, method="REML", subs et=(goals_3=="0"), data=Bfinal, slab=paste(id)) #modelgs.2<-rma(measure="SMD", yi=es, vi=var, weights=wt, method="REML", subs et=(goals_3=="1"), data=Bfinal, slab=paste(id)) #modelgs.3<-rma(measure="SMD", yi=es, vi=var, weights=wt, mods=~goals_3, meth od="REML", data=Bfinal)

#summary(modelgs.1)

255

#summary(modelgs.2) #summary(modelgs.3)

Sub-Group Observation modelobs.1<-rma(measure="SMD", yi=es, vi=var, weights=wt, method="REML", subs et=(observ_4=="0"), data=Bfinal, slab=paste(id)) modelobs.2<-rma(measure="SMD", yi=es, vi=var, weights=wt, method="REML", subs et=(observ_4=="1"), data=Bfinal, slab=paste(id)) modelobs.3<-rma(measure="SMD", yi=es, vi=var, weights=wt, mods=~observ_4, met hod="REML", data=Bfinal)

summary(modelobs.1)

## ## Random-Effects Model (k = 11; tau^2 estimator: REML) ## ## logLik deviance AIC BIC AICc ## -8.8270 17.6540 21.6540 22.2592 23.3683 ## ## tau^2 (estimated amount of total heterogeneity): 0.2599 (SE = 0.1330) ## tau (square root of estimated tau^2 value): 0.5098 ## I^2 (total heterogeneity / total variability): 91.69% ## H^2 (total variability / sampling variability): 12.03 ## ## Test for Heterogeneity: ## Q(df = 10) = 125.5528, p-val < .0001 ## ## Model Results: ## ## estimate se zval pval ci.lb ci.ub ## 0.4787 0.2523 1.8976 0.0577 -0.0157 0.9731 . ## ## --- ## Signif. codes: 0 '***' 0.001 '**' 0.01 '*' 0.05 '.' 0.1 ' ' 1 summary(modelobs.2)

## ## Fixed-Effects Model (k = 1) ## ## logLik deviance AIC BIC AICc ## 0.8897 0.0000 0.2207 -1.7793 4.2207 ## ## Test for Heterogeneity: ## Q(df = 0) = 0.0000, p-val = 1.0000 ## ## Model Results: ## ## estimate se zval pval ci.lb ci.ub

256

## 0.4060 0.1639 2.4773 0.0132 0.0848 0.7272 * ## ## --- ## Signif. codes: 0 '***' 0.001 '**' 0.01 '*' 0.05 '.' 0.1 ' ' 1 summary(modelobs.3)

## ## Mixed-Effects Model (k = 12; tau^2 estimator: REML) ## ## logLik deviance AIC BIC AICc ## -8.8270 17.6540 23.6540 24.5618 27.6540 ## ## tau^2 (estimated amount of residual heterogeneity): 0.2599 (SE = 0.1330) ## tau (square root of estimated tau^2 value): 0.5098 ## I^2 (residual heterogeneity / unaccounted variability): 91.69% ## H^2 (unaccounted variability / sampling variability): 12.03 ## R^2 (amount of heterogeneity accounted for): 0.00% ## ## Test for Residual Heterogeneity: ## QE(df = 10) = 125.5528, p-val < .0001 ## ## Test of Moderators (coefficient(s) 2): ## QM(df = 1) = 0.0151, p-val = 0.9023 ## ## Model Results: ## ## estimate se zval pval ci.lb ci.ub ## intrcpt 0.4787 0.2523 1.8976 0.0577 -0.0157 0.9731 . ## observ_4 -0.0727 0.5919 -0.1228 0.9023 -1.2328 1.0874 ## ## --- ## Signif. codes: 0 '***' 0.001 '**' 0.01 '*' 0.05 '.' 0.1 ' ' 1

Sub-Group Reflection/Self-Evaluation modelref.1<-rma(measure="SMD", yi=es, vi=var, weights=wt, method="REML", subs et=(reflect_5=="0"), data=Bfinal, slab=paste(id)) modelref.2<-rma(measure="SMD", yi=es, vi=var, weights=wt, method="REML", subs et=(reflect_5=="1"), data=Bfinal, slab=paste(id)) modelref.3<-rma(measure="SMD", yi=es, vi=var, weights=wt, mods=~reflect_5, me thod="REML", data=Bfinal)

summary(modelref.1)

## ## Random-Effects Model (k = 11; tau^2 estimator: REML) ## ## logLik deviance AIC BIC AICc ## -8.5049 17.0097 21.0097 21.6149 22.7240

257

## ## tau^2 (estimated amount of total heterogeneity): 0.2480 (SE = 0.1268) ## tau (square root of estimated tau^2 value): 0.4980 ## I^2 (total heterogeneity / total variability): 91.66% ## H^2 (total variability / sampling variability): 11.99 ## ## Test for Heterogeneity: ## Q(df = 10) = 120.7643, p-val < .0001 ## ## Model Results: ## ## estimate se zval pval ci.lb ci.ub ## 0.4914 0.2410 2.0388 0.0415 0.0190 0.9639 * ## ## --- ## Signif. codes: 0 '***' 0.001 '**' 0.01 '*' 0.05 '.' 0.1 ' ' 1 summary(modelref.2)

## ## Fixed-Effects Model (k = 1) ## ## logLik deviance AIC BIC AICc ## 0.5930 0.0000 0.8140 -1.1860 4.8140 ## ## Test for Heterogeneity: ## Q(df = 0) = 0.0000, p-val = 1.0000 ## ## Model Results: ## ## estimate se zval pval ci.lb ci.ub ## -0.0090 0.2205 -0.0407 0.9676 -0.4411 0.4232 ## ## --- ## Signif. codes: 0 '***' 0.001 '**' 0.01 '*' 0.05 '.' 0.1 ' ' 1 summary(modelref.3)

## ## Mixed-Effects Model (k = 12; tau^2 estimator: REML) ## ## logLik deviance AIC BIC AICc ## -8.5049 17.0097 23.0097 23.9175 27.0097 ## ## tau^2 (estimated amount of residual heterogeneity): 0.2480 (SE = 0.1268) ## tau (square root of estimated tau^2 value): 0.4980 ## I^2 (residual heterogeneity / unaccounted variability): 91.66% ## H^2 (unaccounted variability / sampling variability): 11.99 ## R^2 (amount of heterogeneity accounted for): 0.00% ## ## Test for Residual Heterogeneity:

258

## QE(df = 10) = 120.7643, p-val < .0001 ## ## Test of Moderators (coefficient(s) 2): ## QM(df = 1) = 0.7059, p-val = 0.4008 ## ## Model Results: ## ## estimate se zval pval ci.lb ci.ub ## intrcpt 0.4914 0.2410 2.0388 0.0415 0.0190 0.9639 * ## reflect_5 -0.5004 0.5956 -0.8402 0.4008 -1.6678 0.6669 ## ## --- ## Signif. codes: 0 '***' 0.001 '**' 0.01 '*' 0.05 '.' 0.1 ' ' 1

Sub-Group Hands-On Learning modelhol.1<-rma(measure="SMD", yi=es, vi=var, weights=wt, method="REML", subs et=(hands_6=="0"), data=Bfinal, slab=paste(id)) modelhol.2<-rma(measure="SMD", yi=es, vi=var, weights=wt, method="REML", subs et=(hands_6=="1"), data=Bfinal, slab=paste(id)) modelhol.3<-rma(measure="SMD", yi=es, vi=var, weights=wt, mods=~hands_6, meth od="REML", data=Bfinal)

summary(modelhol.1)

## ## Random-Effects Model (k = 9; tau^2 estimator: REML) ## ## logLik deviance AIC BIC AICc ## -6.0398 12.0797 16.0797 16.2386 18.4797 ## ## tau^2 (estimated amount of total heterogeneity): 0.2265 (SE = 0.1261) ## tau (square root of estimated tau^2 value): 0.4760 ## I^2 (total heterogeneity / total variability): 92.20% ## H^2 (total variability / sampling variability): 12.83 ## ## Test for Heterogeneity: ## Q(df = 8) = 109.1378, p-val < .0001 ## ## Model Results: ## ## estimate se zval pval ci.lb ci.ub ## 0.5073 0.2390 2.1225 0.0338 0.0389 0.9758 * ## ## --- ## Signif. codes: 0 '***' 0.001 '**' 0.01 '*' 0.05 '.' 0.1 ' ' 1 summary(modelhol.2)

259

## ## Random-Effects Model (k = 3; tau^2 estimator: REML) ## ## logLik deviance AIC BIC AICc ## -1.5800 3.1599 7.1599 4.5462 19.1599 ## ## tau^2 (estimated amount of total heterogeneity): 0.2065 (SE = 0.2849) ## tau (square root of estimated tau^2 value): 0.4544 ## I^2 (total heterogeneity / total variability): 72.97% ## H^2 (total variability / sampling variability): 3.70 ## ## Test for Heterogeneity: ## Q(df = 2) = 7.9027, p-val = 0.0192 ## ## Model Results: ## ## estimate se zval pval ci.lb ci.ub ## 0.0337 0.3177 0.1061 0.9155 -0.5889 0.6563 ## ## --- ## Signif. codes: 0 '***' 0.001 '**' 0.01 '*' 0.05 '.' 0.1 ' ' 1 summary(modelhol.3)

## ## Mixed-Effects Model (k = 12; tau^2 estimator: REML) ## ## logLik deviance AIC BIC AICc ## -7.6236 15.2472 21.2472 22.1549 25.2472 ## ## tau^2 (estimated amount of residual heterogeneity): 0.2235 (SE = 0.1151) ## tau (square root of estimated tau^2 value): 0.4727 ## I^2 (residual heterogeneity / unaccounted variability): 90.84% ## H^2 (unaccounted variability / sampling variability): 10.92 ## R^2 (amount of heterogeneity accounted for): 3.94% ## ## Test for Residual Heterogeneity: ## QE(df = 10) = 117.0405, p-val < .0001 ## ## Test of Moderators (coefficient(s) 2): ## QM(df = 1) = 1.3711, p-val = 0.2416 ## ## Model Results: ## ## estimate se zval pval ci.lb ci.ub ## intrcpt 0.5073 0.2374 2.1367 0.0326 0.0420 0.9727 * ## hands_6 -0.4737 0.4045 -1.1710 0.2416 -1.2665 0.3192 ## ## --- ## Signif. codes: 0 '***' 0.001 '**' 0.01 '*' 0.05 '.' 0.1 ' ' 1

260

Sub-Group Work Sample Analysis modelws.1<-rma(measure="SMD", yi=es, vi=var, weights=wt, method="REML", subse t=(wsan_7=="0"), data=Bfinal, slab=paste(id)) modelws.2<-rma(measure="SMD", yi=es, vi=var, weights=wt, method="REML", subse t=(wsan_7=="1"), data=Bfinal, slab=paste(id)) modelws.3<-rma(measure="SMD", yi=es, vi=var, weights=wt, mods=~wsan_7, method ="REML", data=Bfinal)

summary(modelws.1)

## ## Random-Effects Model (k = 10; tau^2 estimator: REML) ## ## logLik deviance AIC BIC AICc ## -8.1792 16.3585 20.3585 20.7529 22.3585 ## ## tau^2 (estimated amount of total heterogeneity): 0.2807 (SE = 0.1497) ## tau (square root of estimated tau^2 value): 0.5298 ## I^2 (total heterogeneity / total variability): 92.57% ## H^2 (total variability / sampling variability): 13.46 ## ## Test for Heterogeneity: ## Q(df = 9) = 120.4735, p-val < .0001 ## ## Model Results: ## ## estimate se zval pval ci.lb ci.ub ## 0.4974 0.2711 1.8349 0.0665 -0.0339 1.0286 . ## ## --- ## Signif. codes: 0 '***' 0.001 '**' 0.01 '*' 0.05 '.' 0.1 ' ' 1 summary(modelws.2)

## ## Random-Effects Model (k = 2; tau^2 estimator: REML) ## ## logLik deviance AIC BIC AICc ## -0.2061 0.4122 4.4122 0.4122 16.4122 ## ## tau^2 (estimated amount of total heterogeneity): 0.0484 (SE = 0.1218) ## tau (square root of estimated tau^2 value): 0.2199 ## I^2 (total heterogeneity / total variability): 56.17% ## H^2 (total variability / sampling variability): 2.28 ## ## Test for Heterogeneity: ## Q(df = 1) = 2.2816, p-val = 0.1309 ## ## Model Results:

261

## ## estimate se zval pval ci.lb ci.ub ## 0.2583 0.2085 1.2387 0.2155 -0.1504 0.6670 ## ## --- ## Signif. codes: 0 '***' 0.001 '**' 0.01 '*' 0.05 '.' 0.1 ' ' 1 summary(modelws.3)

## ## Mixed-Effects Model (k = 12; tau^2 estimator: REML) ## ## logLik deviance AIC BIC AICc ## -8.6887 17.3774 23.3774 24.2852 27.3774 ## ## tau^2 (estimated amount of residual heterogeneity): 0.2573 (SE = 0.1315) ## tau (square root of estimated tau^2 value): 0.5073 ## I^2 (residual heterogeneity / unaccounted variability): 91.64% ## H^2 (unaccounted variability / sampling variability): 11.96 ## R^2 (amount of heterogeneity accounted for): 0.00% ## ## Test for Residual Heterogeneity: ## QE(df = 10) = 122.7551, p-val < .0001 ## ## Test of Moderators (coefficient(s) 2): ## QM(df = 1) = 0.2550, p-val = 0.6136 ## ## Model Results: ## ## estimate se zval pval ci.lb ci.ub ## intrcpt 0.4974 0.2598 1.9144 0.0556 -0.0118 1.0066 . ## wsan_7 -0.2391 0.4734 -0.5049 0.6136 -1.1670 0.6889 ## ## --- ## Signif. codes: 0 '***' 0.001 '**' 0.01 '*' 0.05 '.' 0.1 ' ' 1

Sub-Group Lesson Planning modellp.1<-rma(measure="SMD", yi=es, vi=var, weights=wt, method="REML", subse t=(lps_8=="0"), data=Bfinal, slab=paste(id)) modellp.2<-rma(measure="SMD", yi=es, vi=var, weights=wt, method="REML", subse t=(lps_8=="1"), data=Bfinal, slab=paste(id)) modellp.3<-rma(measure="SMD", yi=es, vi=var, weights=wt, mods=~lps_8, method= "REML", data=Bfinal)

summary(modellp.1)

## ## Random-Effects Model (k = 8; tau^2 estimator: REML) ##

262

## logLik deviance AIC BIC AICc ## -6.2555 12.5111 16.5111 16.4029 19.5111 ## ## tau^2 (estimated amount of total heterogeneity): 0.3040 (SE = 0.1815) ## tau (square root of estimated tau^2 value): 0.5513 ## I^2 (total heterogeneity / total variability): 90.95% ## H^2 (total variability / sampling variability): 11.06 ## ## Test for Heterogeneity: ## Q(df = 7) = 74.9111, p-val < .0001 ## ## Model Results: ## ## estimate se zval pval ci.lb ci.ub ## 0.2962 0.2214 1.3374 0.1811 -0.1379 0.7302 ## ## --- ## Signif. codes: 0 '***' 0.001 '**' 0.01 '*' 0.05 '.' 0.1 ' ' 1 summary(modellp.2)

## ## Random-Effects Model (k = 4; tau^2 estimator: REML) ## ## logLik deviance AIC BIC AICc ## -3.1648 6.3295 10.3295 8.5268 22.3295 ## ## tau^2 (estimated amount of total heterogeneity): 0.1659 (SE = 0.1672) ## tau (square root of estimated tau^2 value): 0.4074 ## I^2 (total heterogeneity / total variability): 87.17% ## H^2 (total variability / sampling variability): 7.79 ## ## Test for Heterogeneity: ## Q(df = 3) = 34.9206, p-val < .0001 ## ## Model Results: ## ## estimate se zval pval ci.lb ci.ub ## 0.6245 0.3252 1.9205 0.0548 -0.0128 1.2619 . ## ## --- ## Signif. codes: 0 '***' 0.001 '**' 0.01 '*' 0.05 '.' 0.1 ' ' 1 summary(modellp.3)

## ## Mixed-Effects Model (k = 12; tau^2 estimator: REML) ## ## logLik deviance AIC BIC AICc ## -9.1960 18.3920 24.3920 25.2998 28.3920 ##

263

## tau^2 (estimated amount of residual heterogeneity): 0.2582 (SE = 0.1320) ## tau (square root of estimated tau^2 value): 0.5081 ## I^2 (residual heterogeneity / unaccounted variability): 90.15% ## H^2 (unaccounted variability / sampling variability): 10.15 ## R^2 (amount of heterogeneity accounted for): 0.00% ## ## Test for Residual Heterogeneity: ## QE(df = 10) = 109.8317, p-val < .0001 ## ## Test of Moderators (coefficient(s) 2): ## QM(df = 1) = 0.5260, p-val = 0.4683 ## ## Model Results: ## ## estimate se zval pval ci.lb ci.ub ## intrcpt 0.2962 0.2054 1.4416 0.1494 -0.1065 0.6988 ## lps_8 0.3284 0.4528 0.7252 0.4683 -0.5590 1.2158 ## ## --- ## Signif. codes: 0 '***' 0.001 '**' 0.01 '*' 0.05 '.' 0.1 ' ' 1

Sub-Group Study quality modqs.1<-rma(yi=es, vi=var, weights=wt, method="REML", subset=(qs_rank=="low" ), data=Bfinal, slab=paste(id)) modqs.2<-rma(yi=es, vi=var, weights=wt, method="REML", subset=(qs_rank=="ave" ), data=Bfinal, slab=paste(id)) modqs.3<-rma(yi=es, vi=var, weights=wt, method="REML", subset=(qs_rank=="high "), data=Bfinal, slab=paste(id)) modqs.4<-rma(measure="SMD", yi=es, vi=var, weights=wt, mods=~qs_rank, method= "REML", data=Bfinal)

summary(modqs.1)

## ## Random-Effects Model (k = 3; tau^2 estimator: REML) ## ## logLik deviance AIC BIC AICc ## -2.2507 4.5014 8.5014 5.8877 20.5014 ## ## tau^2 (estimated amount of total heterogeneity): 0.4841 (SE = 0.5152) ## tau (square root of estimated tau^2 value): 0.6958 ## I^2 (total heterogeneity / total variability): 94.46% ## H^2 (total variability / sampling variability): 18.07 ## ## Test for Heterogeneity: ## Q(df = 2) = 44.8441, p-val < .0001 ## ## Model Results:

264

## ## estimate se zval pval ci.lb ci.ub ## 0.2510 0.4531 0.5540 0.5796 -0.6370 1.1390 ## ## --- ## Signif. codes: 0 '***' 0.001 '**' 0.01 '*' 0.05 '.' 0.1 ' ' 1 summary(modqs.2)

## ## Random-Effects Model (k = 5; tau^2 estimator: REML) ## ## logLik deviance AIC BIC AICc ## -3.1120 6.2239 10.2239 8.9965 22.2239 ## ## tau^2 (estimated amount of total heterogeneity): 0.1845 (SE = 0.1549) ## tau (square root of estimated tau^2 value): 0.4295 ## I^2 (total heterogeneity / total variability): 89.32% ## H^2 (total variability / sampling variability): 9.37 ## ## Test for Heterogeneity: ## Q(df = 4) = 35.4932, p-val < .0001 ## ## Model Results: ## ## estimate se zval pval ci.lb ci.ub ## 0.6668 0.3081 2.1640 0.0305 0.0629 1.2707 * ## ## --- ## Signif. codes: 0 '***' 0.001 '**' 0.01 '*' 0.05 '.' 0.1 ' ' 1 summary(modqs.3)

## ## Random-Effects Model (k = 4; tau^2 estimator: REML) ## ## logLik deviance AIC BIC AICc ## -1.5432 3.0865 7.0865 5.2837 19.0865 ## ## tau^2 (estimated amount of total heterogeneity): 0.0952 (SE = 0.1120) ## tau (square root of estimated tau^2 value): 0.3085 ## I^2 (total heterogeneity / total variability): 71.48% ## H^2 (total variability / sampling variability): 3.51 ## ## Test for Heterogeneity: ## Q(df = 3) = 9.4173, p-val = 0.0242 ## ## Model Results: ## ## estimate se zval pval ci.lb ci.ub ## 0.0906 0.1936 0.4683 0.6396 -0.2887 0.4700

265

## ## --- ## Signif. codes: 0 '***' 0.001 '**' 0.01 '*' 0.05 '.' 0.1 ' ' 1 summary(modqs.4)

## ## Mixed-Effects Model (k = 12; tau^2 estimator: REML) ## ## logLik deviance AIC BIC AICc ## -7.4913 14.9826 22.9826 23.7715 32.9826 ## ## tau^2 (estimated amount of residual heterogeneity): 0.2345 (SE = 0.1280) ## tau (square root of estimated tau^2 value): 0.4842 ## I^2 (residual heterogeneity / unaccounted variability): 89.60% ## H^2 (unaccounted variability / sampling variability): 9.62 ## R^2 (amount of heterogeneity accounted for): 0.00% ## ## Test for Residual Heterogeneity: ## QE(df = 9) = 89.7546, p-val < .0001 ## ## Test of Moderators (coefficient(s) 2:3): ## QM(df = 2) = 1.6974, p-val = 0.4280 ## ## Model Results: ## ## estimate se zval pval ci.lb ci.ub ## intrcpt 0.6668 0.3463 1.9256 0.0542 -0.0119 1.3455 . ## qs_rankhigh -0.5762 0.4464 -1.2907 0.1968 -1.4511 0.2988 ## qs_ranklow -0.4158 0.4729 -0.8792 0.3793 -1.3428 0.5112 ## ## --- ## Signif. codes: 0 '***' 0.001 '**' 0.01 '*' 0.05 '.' 0.1 ' ' 1

Sub-Group Validity modelv.1<-rma(yi=es, vi=var, weights=wt, method="REML", subset=(valid=="SST") , data=Bfinal, slab=paste(id)) modelv.2<-rma(yi=es, vi=var, weights=wt, method="REML", subset=(valid=="PVM") , data=Bfinal, slab=paste(id)) modelv.3<-rma(yi=es, vi=var, weights=wt, method="REML", subset=(valid=="PVWOM "), data=Bfinal, slab=paste(id)) modelv.4<-rma(measure="SMD", yi=es, vi=var, weights=wt, mods=~valid, method=" REML", data=Bfinal)

summary(modelv.1)

## ## Random-Effects Model (k = 3; tau^2 estimator: REML) ##

266

## logLik deviance AIC BIC AICc ## -1.7551 3.5103 7.5103 4.8966 19.5103 ## ## tau^2 (estimated amount of total heterogeneity): 0.2914 (SE = 0.3277) ## tau (square root of estimated tau^2 value): 0.5398 ## I^2 (total heterogeneity / total variability): 89.14% ## H^2 (total variability / sampling variability): 9.21 ## ## Test for Heterogeneity: ## Q(df = 2) = 17.5050, p-val = 0.0002 ## ## Model Results: ## ## estimate se zval pval ci.lb ci.ub ## 0.6428 0.3357 1.9151 0.0555 -0.0151 1.3008 . ## ## --- ## Signif. codes: 0 '***' 0.001 '**' 0.01 '*' 0.05 '.' 0.1 ' ' 1 summary(modelv.2)

## ## Random-Effects Model (k = 5; tau^2 estimator: REML) ## ## logLik deviance AIC BIC AICc ## -1.4782 2.9565 6.9565 5.7291 18.9565 ## ## tau^2 (estimated amount of total heterogeneity): 0.0621 (SE = 0.0735) ## tau (square root of estimated tau^2 value): 0.2493 ## I^2 (total heterogeneity / total variability): 62.88% ## H^2 (total variability / sampling variability): 2.69 ## ## Test for Heterogeneity: ## Q(df = 4) = 10.0312, p-val = 0.0399 ## ## Model Results: ## ## estimate se zval pval ci.lb ci.ub ## 0.0541 0.1508 0.3588 0.7197 -0.2414 0.3496 ## ## --- ## Signif. codes: 0 '***' 0.001 '**' 0.01 '*' 0.05 '.' 0.1 ' ' 1 summary(modelv.3)

## ## Random-Effects Model (k = 4; tau^2 estimator: REML) ## ## logLik deviance AIC BIC AICc ## -2.6060 5.2120 9.2120 7.4093 21.2120 ##

267

## tau^2 (estimated amount of total heterogeneity): 0.2938 (SE = 0.2604) ## tau (square root of estimated tau^2 value): 0.5420 ## I^2 (total heterogeneity / total variability): 94.51% ## H^2 (total variability / sampling variability): 18.21 ## ## Test for Heterogeneity: ## Q(df = 3) = 63.7867, p-val < .0001 ## ## Model Results: ## ## estimate se zval pval ci.lb ci.ub ## 0.6041 0.3921 1.5408 0.1234 -0.1644 1.3726 ## ## --- ## Signif. codes: 0 '***' 0.001 '**' 0.01 '*' 0.05 '.' 0.1 ' ' 1 summary(modelv.4)

## ## Mixed-Effects Model (k = 12; tau^2 estimator: REML) ## ## logLik deviance AIC BIC AICc ## -6.5257 13.0514 21.0514 21.8403 31.0514 ## ## tau^2 (estimated amount of residual heterogeneity): 0.2041 (SE = 0.1136) ## tau (square root of estimated tau^2 value): 0.4518 ## I^2 (residual heterogeneity / unaccounted variability): 88.55% ## H^2 (unaccounted variability / sampling variability): 8.74 ## R^2 (amount of heterogeneity accounted for): 12.25% ## ## Test for Residual Heterogeneity: ## QE(df = 9) = 91.3229, p-val < .0001 ## ## Test of Moderators (coefficient(s) 2:3): ## QM(df = 2) = 3.1011, p-val = 0.2121 ## ## Model Results: ## ## estimate se zval pval ci.lb ci.ub ## intrcpt 0.0541 0.2431 0.2226 0.8238 -0.4223 0.5305 ## validPVWOM 0.5500 0.4083 1.3470 0.1780 -0.2503 1.3503 ## validSST 0.5887 0.3762 1.5652 0.1175 -0.1485 1.3260 ## ## --- ## Signif. codes: 0 '***' 0.001 '**' 0.01 '*' 0.05 '.' 0.1 ' ' 1

Sub-Group Reliability modelrel.1<-rma(measure="SMD", yi=es, vi=var, weights=wt, method="REML", subs et=(reliable=="0"), data=Bfinal, slab=paste(id)) modelrel.2<-rma(measure="SMD", yi=es, vi=var, weights=wt, method="REML", subs

268

et=(reliable=="1"), data=Bfinal, slab=paste(id)) modelrel.3<-rma(measure="SMD", yi=es, vi=var, weights=wt, mods=~reliable, met hod="REML", data=Bfinal)

summary(modelrel.1)

## ## Random-Effects Model (k = 7; tau^2 estimator: REML) ## ## logLik deviance AIC BIC AICc ## -5.7724 11.5447 15.5447 15.1282 19.5447 ## ## tau^2 (estimated amount of total heterogeneity): 0.2299 (SE = 0.1576) ## tau (square root of estimated tau^2 value): 0.4795 ## I^2 (total heterogeneity / total variability): 89.63% ## H^2 (total variability / sampling variability): 9.64 ## ## Test for Heterogeneity: ## Q(df = 6) = 48.5375, p-val < .0001 ## ## Model Results: ## ## estimate se zval pval ci.lb ci.ub ## 0.6298 0.3173 1.9848 0.0472 0.0079 1.2516 * ## ## --- ## Signif. codes: 0 '***' 0.001 '**' 0.01 '*' 0.05 '.' 0.1 ' ' 1 summary(modelrel.2)

## ## Random-Effects Model (k = 5; tau^2 estimator: REML) ## ## logLik deviance AIC BIC AICc ## -3.3935 6.7871 10.7871 9.5597 22.7871 ## ## tau^2 (estimated amount of total heterogeneity): 0.2857 (SE = 0.2219) ## tau (square root of estimated tau^2 value): 0.5345 ## I^2 (total heterogeneity / total variability): 91.73% ## H^2 (total variability / sampling variability): 12.09 ## ## Test for Heterogeneity: ## Q(df = 4) = 48.9890, p-val < .0001 ## ## Model Results: ## ## estimate se zval pval ci.lb ci.ub ## 0.1691 0.2639 0.6407 0.5217 -0.3482 0.6863 ##

269

## --- ## Signif. codes: 0 '***' 0.001 '**' 0.01 '*' 0.05 '.' 0.1 ' ' 1 summary(modelrel.3)

## ## Mixed-Effects Model (k = 12; tau^2 estimator: REML) ## ## logLik deviance AIC BIC AICc ## -9.1272 18.2543 24.2543 25.1621 28.2543 ## ## tau^2 (estimated amount of residual heterogeneity): 0.2547 (SE = 0.1305) ## tau (square root of estimated tau^2 value): 0.5047 ## I^2 (residual heterogeneity / unaccounted variability): 90.65% ## H^2 (unaccounted variability / sampling variability): 10.70 ## R^2 (amount of heterogeneity accounted for): 0.00% ## ## Test for Residual Heterogeneity: ## QE(df = 10) = 97.5265, p-val < .0001 ## ## Test of Moderators (coefficient(s) 2): ## QM(df = 1) = 1.2203, p-val = 0.2693 ## ## Model Results: ## ## estimate se zval pval ci.lb ci.ub ## intrcpt 0.6298 0.3336 1.8880 0.0590 -0.0240 1.2835 . ## reliable -0.4607 0.4170 -1.1047 0.2693 -1.2780 0.3567 ## ## --- ## Signif. codes: 0 '***' 0.001 '**' 0.01 '*' 0.05 '.' 0.1 ' ' 1

270

APPENDIX M R CODE MARKDOWN DOCUMENT - KNOWLEDGE

Data Summary, Main Effect Analyses, and Model Fit Comparison

#read data Kfinal <- read.csv("Agg_Know2_pop.csv") summary(Kfinal)

## study id n ## Abbitt (2011) : 1 AU56 : 1 Min. : 7.00 ## Agyei & Keengwe (2014) : 1 CB73 : 1 1st Qu.: 22.25 ## Agyei & Voogt (2015) : 1 CM33 : 1 Median : 45.00 ## Alayyar et al. (2012) : 1 CV48_1 : 1 Mean : 71.11 ## Alexander et al. (2014): 1 CV48_2 : 1 3rd Qu.: 77.50 ## An et al. (2011) : 1 DD95 : 1 Max. :365.00 ## (Other) :40 (Other):40 ## es var wt reliable ## Min. :-0.7794 Min. :0.004623 Min. : 1.351 Min. :0.0 ## 1st Qu.: 0.5669 1st Qu.:0.031735 1st Qu.: 10.553 1st Qu.:0.0 ## Median : 0.9391 Median :0.047283 Median : 21.150 Median :0.5 ## Mean : 1.1150 Mean :0.089871 Mean : 34.235 Mean :0.5 ## 3rd Qu.: 1.5998 3rd Qu.:0.094770 3rd Qu.: 31.512 3rd Qu.:1.0 ## Max. : 3.5669 Max. :0.740097 Max. :216.315 Max. :1.0 ## ## valid self_report mentor_1 fexp_2 ## PVM :16 Min. :0.0000 Min. :0.0000 Min. :0.0000 ## PVWOM:19 1st Qu.:1.0000 1st Qu.:0.0000 1st Qu.:0.0000 ## SST :11 Median :1.0000 Median :0.0000 Median :0.0000 ## Mean :0.7826 Mean :0.1087 Mean :0.1739 ## 3rd Qu.:1.0000 3rd Qu.:0.0000 3rd Qu.:0.0000 ## Max. :1.0000 Max. :1.0000 Max. :1.0000 ## ## goals_3 observ_4 reflect_5 hands_6 ## Min. :0.00000 Min. :0.0000 Min. :0.0000 Min. :0.0000 ## 1st Qu.:0.00000 1st Qu.:0.0000 1st Qu.:0.0000 1st Qu.:0.0000 ## Median :0.00000 Median :0.0000 Median :0.0000 Median :0.0000 ## Mean :0.02174 Mean :0.1304 Mean :0.3261 Mean :0.3261 ## 3rd Qu.:0.00000 3rd Qu.:0.0000 3rd Qu.:1.0000 3rd Qu.:1.0000 ## Max. :1.00000 Max. :1.0000 Max. :1.0000 Max. :1.0000 ## ## wsan_7 lps_8 q_score qs_rank ## Min. :0.0000 Min. :0.000 Min. : 6.00 ave :26 ## 1st Qu.:0.0000 1st Qu.:0.000 1st Qu.:14.00 high:11 ## Median :0.0000 Median :1.000 Median :16.50 low : 9 ## Mean :0.2174 Mean :0.587 Mean :16.43 ## 3rd Qu.:0.0000 3rd Qu.:1.000 3rd Qu.:19.00 ## Max. :1.0000 Max. :1.000 Max. :26.00 ##

271

#Fixed-Effect Modeling library(metafor)

## Warning: package 'metafor' was built under R version 3.4.1

## Loading required package: Matrix

## Warning: package 'Matrix' was built under R version 3.4.2

summary(model1)

## ## Fixed-Effects Model (k = 46) ## ## logLik deviance AIC BIC AICc ## -244.9169 542.1600 491.8338 493.6625 491.9247 ## ## Test for Heterogeneity: ## Q(df = 45) = 542.1600, p-val < .0001 ## ## Model Results: ## ## estimate se zval pval ci.lb ci.ub ## 0.8540 0.0252 33.8889 <.0001 0.8046 0.9034 *** ## ## --- ## Signif. codes: 0 '***' 0.001 '**' 0.01 '*' 0.05 '.' 0.1 ' ' 1

#Random-Effects Modeling model2<-rma(measure="SMD", yi=es, vi=var, weights=wt, method="REML", data=Kfi nal,slab=paste(study)) summary(model2)

## ## Random-Effects Model (k = 46; tau^2 estimator: REML) ## ## logLik deviance AIC BIC AICc ## -55.1157 110.2313 114.2313 117.8447 114.5171 ## ## tau^2 (estimated amount of total heterogeneity): 0.5111 (SE = 0.1222) ## tau (square root of estimated tau^2 value): 0.7149 ## I^2 (total heterogeneity / total variability): 94.38%

272

## H^2 (total variability / sampling variability): 17.79 ## ## Test for Heterogeneity: ## Q(df = 45) = 542.1600, p-val < .0001 ## ## Model Results: ## ## estimate se zval pval ci.lb ci.ub ## 0.8540 0.1788 4.7750 <.0001 0.5034 1.2045 *** ## ## --- ## Signif. codes: 0 '***' 0.001 '**' 0.01 '*' 0.05 '.' 0.1 ' ' 1

#Model Fit Assessment anova(model2, model1)

## df AIC BIC AICc logLik LRT pval QE ## Full 2 114.2313 117.8447 114.5171 -55.1157 542.1600 ## Reduced 1 493.5292 495.3359 493.6222 -245.7646 381.2978 <.0001 542.1600 ## tau^2 R^2 ## Full 0.5111 ## Reduced 0.0000 NA%

Forrest Plot forest(model2, digits = 3, order = "obs") text(-9, 49, "Author(s) and Year", pos=4) text(13, 49, "Std. Dev and 95% C.I.", pos=2)

Bias Check - Fail-Safe N fsn(yi=es, vi=var, data=Kfinal, type ="Rosenthal")

## ## Fail-safe N Calculation Using the Rosenthal Approach ## ## Observed Significance Level: <.0001 ## Target Significance Level: 0.05 ## ## Fail-safe N: 18819 fsn(yi=es, vi=var, data=Kfinal, type ="Rosenberg")

## ## Fail-safe N Calculation Using the Rosenberg Approach ## ## Average Effect Size: 0.8540 ## Observed Significance Level: <.0001 ## Target Significance Level: 0.05

273

## ## Fail-safe N: 13707 fsn(yi=es, vi=var, data=Kfinal, type ="Orwin", target=)

## ## Fail-safe N Calculation Using the Orwin Approach ## ## Average Effect Size: 1.1150 ## Target Effect Size: 0.5575 ## ## Fail-safe N: 46

Bias Check - Trim-and-Fill Plot library(metafor)

### carry out trim-and-fill analysis taf <- trimfill(model2)

### draw funnel plot with missing studies filled in funnel(taf)

Sub-Group Analyses Sub-Group Mentoring/Coaching modelmen.1<-rma(measure="SMD", yi=es, vi=var, weights=wt, method="REML", subs et=(mentor_1=="0"), data=Kfinal, slab=paste(id)) modelmen.2<-rma(measure="SMD", yi=es, vi=var, weights=wt, method="REML", subs et=(mentor_1=="1"), data=Kfinal, slab=paste(id)) modelmen.3<-rma(measure="SMD", yi=es, vi=var, weights=wt, mods=~mentor_1, met hod="REML", data=Kfinal)

summary(modelmen.1)

## ## Random-Effects Model (k = 41; tau^2 estimator: REML) ## ## logLik deviance AIC BIC AICc ## -50.6033 101.2066 105.2066 108.5843 105.5309 ## ## tau^2 (estimated amount of total heterogeneity): 0.5617 (SE = 0.1414) ## tau (square root of estimated tau^2 value): 0.7495 ## I^2 (total heterogeneity / total variability): 94.87% ## H^2 (total variability / sampling variability): 19.51 ## ## Test for Heterogeneity: ## Q(df = 40) = 512.4642, p-val < .0001 ##

274

## Model Results: ## ## estimate se zval pval ci.lb ci.ub ## 0.8645 0.2019 4.2827 <.0001 0.4689 1.2602 *** ## ## --- ## Signif. codes: 0 '***' 0.001 '**' 0.01 '*' 0.05 '.' 0.1 ' ' 1 summary(modelmen.2)

## ## Random-Effects Model (k = 5; tau^2 estimator: REML) ## ## logLik deviance AIC BIC AICc ## -3.4558 6.9116 10.9116 9.6842 22.9116 ## ## tau^2 (estimated amount of total heterogeneity): 0.2018 (SE = 0.1823) ## tau (square root of estimated tau^2 value): 0.4492 ## I^2 (total heterogeneity / total variability): 83.56% ## H^2 (total variability / sampling variability): 6.08 ## ## Test for Heterogeneity: ## Q(df = 4) = 28.1013, p-val < .0001 ## ## Model Results: ## ## estimate se zval pval ci.lb ci.ub ## 0.7579 0.2788 2.7187 0.0066 0.2115 1.3044 ** ## ## --- ## Signif. codes: 0 '***' 0.001 '**' 0.01 '*' 0.05 '.' 0.1 ' ' 1 summary(modelmen.3)

## ## Mixed-Effects Model (k = 46; tau^2 estimator: REML) ## ## logLik deviance AIC BIC AICc ## -54.4470 108.8941 114.8941 120.2467 115.4941 ## ## tau^2 (estimated amount of residual heterogeneity): 0.5254 (SE = 0.1268) ## tau (square root of estimated tau^2 value): 0.7249 ## I^2 (residual heterogeneity / unaccounted variability): 94.43% ## H^2 (unaccounted variability / sampling variability): 17.94 ## R^2 (amount of heterogeneity accounted for): 0.00% ## ## Test for Residual Heterogeneity: ## QE(df = 44) = 540.5656, p-val < .0001 ## ## Test of Moderators (coefficient(s) 2): ## QM(df = 1) = 0.0493, p-val = 0.8242

275

## ## Model Results: ## ## estimate se zval pval ci.lb ci.ub ## intrcpt 0.8645 0.1953 4.4255 <.0001 0.4816 1.2474 *** ## mentor_1 -0.1066 0.4798 -0.2221 0.8242 -1.0470 0.8338 ## ## --- ## Signif. codes: 0 '***' 0.001 '**' 0.01 '*' 0.05 '.' 0.1 ' ' 1

Sub-Group Field Experience/Rehersal modelfe.1<-rma(measure="SMD", yi=es, vi=var, weights=wt, method="REML", subse t=(fexp_2=="0"), data=Kfinal, slab=paste(id)) modelfe.2<-rma(measure="SMD", yi=es, vi=var, weights=wt, method="REML", subse t=(fexp_2=="1"), data=Kfinal, slab=paste(id)) modelfe.3<-rma(measure="SMD", yi=es, vi=var, weights=wt, mods=~fexp_2, method ="REML", data=Kfinal)

summary(modelfe.1)

## ## Random-Effects Model (k = 38; tau^2 estimator: REML) ## ## logLik deviance AIC BIC AICc ## -44.1169 88.2337 92.2337 95.4555 92.5866 ## ## tau^2 (estimated amount of total heterogeneity): 0.5382 (SE = 0.1398) ## tau (square root of estimated tau^2 value): 0.7336 ## I^2 (total heterogeneity / total variability): 94.12% ## H^2 (total variability / sampling variability): 17.00 ## ## Test for Heterogeneity: ## Q(df = 37) = 484.2337, p-val < .0001 ## ## Model Results: ## ## estimate se zval pval ci.lb ci.ub ## 0.9056 0.1947 4.6513 <.0001 0.5240 1.2872 *** ## ## --- ## Signif. codes: 0 '***' 0.001 '**' 0.01 '*' 0.05 '.' 0.1 ' ' 1 summary(modelfe.2)

## ## Random-Effects Model (k = 8; tau^2 estimator: REML) ## ## logLik deviance AIC BIC AICc ## -10.5085 21.0171 25.0171 24.9089 28.0171

276

## ## tau^2 (estimated amount of total heterogeneity): 0.4220 (SE = 0.2771) ## tau (square root of estimated tau^2 value): 0.6496 ## I^2 (total heterogeneity / total variability): 93.80% ## H^2 (total variability / sampling variability): 16.14 ## ## Test for Heterogeneity: ## Q(df = 7) = 45.3077, p-val < .0001 ## ## Model Results: ## ## estimate se zval pval ci.lb ci.ub ## 0.6989 0.3940 1.7738 0.0761 -0.0734 1.4711 . ## ## --- ## Signif. codes: 0 '***' 0.001 '**' 0.01 '*' 0.05 '.' 0.1 ' ' 1 summary(modelfe.3)

## ## Mixed-Effects Model (k = 46; tau^2 estimator: REML) ## ## logLik deviance AIC BIC AICc ## -54.5861 109.1723 115.1723 120.5249 115.7723 ## ## tau^2 (estimated amount of residual heterogeneity): 0.5276 (SE = 0.1272) ## tau (square root of estimated tau^2 value): 0.7263 ## I^2 (residual heterogeneity / unaccounted variability): 94.19% ## H^2 (unaccounted variability / sampling variability): 17.20 ## R^2 (amount of heterogeneity accounted for): 0.00% ## ## Test for Residual Heterogeneity: ## QE(df = 44) = 529.5414, p-val < .0001 ## ## Test of Moderators (coefficient(s) 2): ## QM(df = 1) = 0.1854, p-val = 0.6668 ## ## Model Results: ## ## estimate se zval pval ci.lb ci.ub ## intrcpt 0.9056 0.1928 4.6967 <.0001 0.5277 1.2836 *** ## fexp_2 -0.2068 0.4802 -0.4306 0.6668 -1.1480 0.7344 ## ## --- ## Signif. codes: 0 '***' 0.001 '**' 0.01 '*' 0.05 '.' 0.1 ' ' 1

Sub-Group Goal-Setting modelgs.1<-rma(measure="SMD", yi=es, vi=var, weights=wt, method="REML", subse t=(goals_3=="0"), data=Kfinal, slab=paste(id)) modelgs.2<-rma(measure="SMD", yi=es, vi=var, weights=wt, method="REML", subse

277

t=(goals_3=="1"), data=Kfinal, slab=paste(id)) modelgs.3<-rma(measure="SMD", yi=es, vi=var, weights=wt, mods=~goals_3, metho d="REML", data=Kfinal)

summary(modelgs.1)

## ## Random-Effects Model (k = 45; tau^2 estimator: REML) ## ## logLik deviance AIC BIC AICc ## -53.2709 106.5417 110.5417 114.1101 110.8344 ## ## tau^2 (estimated amount of total heterogeneity): 0.5138 (SE = 0.1245) ## tau (square root of estimated tau^2 value): 0.7168 ## I^2 (total heterogeneity / total variability): 93.75% ## H^2 (total variability / sampling variability): 16.01 ## ## Test for Heterogeneity: ## Q(df = 44) = 486.9707, p-val < .0001 ## ## Model Results: ## ## estimate se zval pval ci.lb ci.ub ## 0.9275 0.1746 5.3120 <.0001 0.5853 1.2697 *** ## ## --- ## Signif. codes: 0 '***' 0.001 '**' 0.01 '*' 0.05 '.' 0.1 ' ' 1 summary(modelgs.2)

## ## Fixed-Effects Model (k = 1) ## ## logLik deviance AIC BIC AICc ## 1.7560 0.0000 -1.5119 -3.5119 2.4881 ## ## Test for Heterogeneity: ## Q(df = 0) = 0.0000, p-val = 1.0000 ## ## Model Results: ## ## estimate se zval pval ci.lb ci.ub ## 0.3775 0.0689 5.4773 <.0001 0.2424 0.5125 *** ## ## --- ## Signif. codes: 0 '***' 0.001 '**' 0.01 '*' 0.05 '.' 0.1 ' ' 1 summary(modelgs.3)

278

## ## Mixed-Effects Model (k = 46; tau^2 estimator: REML) ## ## logLik deviance AIC BIC AICc ## -53.2709 106.5417 112.5417 117.8943 113.1417 ## ## tau^2 (estimated amount of residual heterogeneity): 0.5138 (SE = 0.1245) ## tau (square root of estimated tau^2 value): 0.7168 ## I^2 (residual heterogeneity / unaccounted variability): 93.75% ## H^2 (unaccounted variability / sampling variability): 16.01 ## R^2 (amount of heterogeneity accounted for): 0.00% ## ## Test for Residual Heterogeneity: ## QE(df = 44) = 486.9707, p-val < .0001 ## ## Test of Moderators (coefficient(s) 2): ## QM(df = 1) = 0.5511, p-val = 0.4579 ## ## Model Results: ## ## estimate se zval pval ci.lb ci.ub ## intrcpt 0.9275 0.1746 5.3120 <.0001 0.5853 1.2697 *** ## goals_3 -0.5501 0.7410 -0.7424 0.4579 -2.0023 0.9022 ## ## --- ## Signif. codes: 0 '***' 0.001 '**' 0.01 '*' 0.05 '.' 0.1 ' ' 1

Sub-Group Observation modelobs.1<-rma(measure="SMD", yi=es, vi=var, weights=wt, method="REML", subs et=(observ_4=="0"), data=Kfinal, slab=paste(id)) modelobs.2<-rma(measure="SMD", yi=es, vi=var, weights=wt, method="REML", subs et=(observ_4=="1"), data=Kfinal, slab=paste(id)) modelobs.3<-rma(measure="SMD", yi=es, vi=var, weights=wt, mods=~observ_4, met hod="REML", data=Kfinal)

summary(modelobs.1)

## ## Random-Effects Model (k = 40; tau^2 estimator: REML) ## ## logLik deviance AIC BIC AICc ## -40.1317 80.2634 84.2634 87.5905 84.5967 ## ## tau^2 (estimated amount of total heterogeneity): 0.3407 (SE = 0.0900) ## tau (square root of estimated tau^2 value): 0.5837 ## I^2 (total heterogeneity / total variability): 92.39% ## H^2 (total variability / sampling variability): 13.15 ##

279

## Test for Heterogeneity: ## Q(df = 39) = 361.3625, p-val < .0001 ## ## Model Results: ## ## estimate se zval pval ci.lb ci.ub ## 0.8168 0.1539 5.3069 <.0001 0.5152 1.1185 *** ## ## --- ## Signif. codes: 0 '***' 0.001 '**' 0.01 '*' 0.05 '.' 0.1 ' ' 1 summary(modelobs.2)

## ## Random-Effects Model (k = 6; tau^2 estimator: REML) ## ## logLik deviance AIC BIC AICc ## -9.1155 18.2310 22.2310 21.4499 28.2310 ## ## tau^2 (estimated amount of total heterogeneity): 2.0013 (SE = 1.3925) ## tau (square root of estimated tau^2 value): 1.4147 ## I^2 (total heterogeneity / total variability): 96.10% ## H^2 (total variability / sampling variability): 25.65 ## ## Test for Heterogeneity: ## Q(df = 5) = 141.9804, p-val < .0001 ## ## Model Results: ## ## estimate se zval pval ci.lb ci.ub ## 1.5178 0.7321 2.0733 0.0381 0.0830 2.9527 * ## ## --- ## Signif. codes: 0 '***' 0.001 '**' 0.01 '*' 0.05 '.' 0.1 ' ' 1 summary(modelobs.3)

## ## Mixed-Effects Model (k = 46; tau^2 estimator: REML) ## ## logLik deviance AIC BIC AICc ## -52.6185 105.2371 111.2371 116.5896 111.8371 ## ## tau^2 (estimated amount of residual heterogeneity): 0.5014 (SE = 0.1214) ## tau (square root of estimated tau^2 value): 0.7081 ## I^2 (residual heterogeneity / unaccounted variability): 94.30% ## H^2 (unaccounted variability / sampling variability): 17.55 ## R^2 (amount of heterogeneity accounted for): 1.90% ## ## Test for Residual Heterogeneity: ## QE(df = 44) = 503.3429, p-val < .0001

280

## ## Test of Moderators (coefficient(s) 2): ## QM(df = 1) = 2.7635, p-val = 0.0964 ## ## Model Results: ## ## estimate se zval pval ci.lb ci.ub ## intrcpt 0.8168 0.1859 4.3945 <.0001 0.4525 1.1812 *** ## observ_4 0.7010 0.4217 1.6624 0.0964 -0.1255 1.5274 . ## ## --- ## Signif. codes: 0 '***' 0.001 '**' 0.01 '*' 0.05 '.' 0.1 ' ' 1

Sub-Group Reflection/Self-Evaluation modelref.1<-rma(measure="SMD", yi=es, vi=var, weights=wt, method="REML", subs et=(reflect_5=="0"), data=Kfinal, slab=paste(id)) modelref.2<-rma(measure="SMD", yi=es, vi=var, weights=wt, method="REML", subs et=(reflect_5=="1"), data=Kfinal, slab=paste(id)) modelref.3<-rma(measure="SMD", yi=es, vi=var, weights=wt, mods=~reflect_5, me thod="REML", data=Kfinal)

summary(modelref.1)

## ## Random-Effects Model (k = 31; tau^2 estimator: REML) ## ## logLik deviance AIC BIC AICc ## -40.4464 80.8929 84.8929 87.6953 85.3373 ## ## tau^2 (estimated amount of total heterogeneity): 0.6120 (SE = 0.1769) ## tau (square root of estimated tau^2 value): 0.7823 ## I^2 (total heterogeneity / total variability): 95.46% ## H^2 (total variability / sampling variability): 22.04 ## ## Test for Heterogeneity: ## Q(df = 30) = 445.1025, p-val < .0001 ## ## Model Results: ## ## estimate se zval pval ci.lb ci.ub ## 0.8385 0.2385 3.5159 0.0004 0.3711 1.3060 *** ## ## --- ## Signif. codes: 0 '***' 0.001 '**' 0.01 '*' 0.05 '.' 0.1 ' ' 1 summary(modelref.2)

## ## Random-Effects Model (k = 15; tau^2 estimator: REML)

281

## ## logLik deviance AIC BIC AICc ## -13.3569 26.7138 30.7138 31.9920 31.8047 ## ## tau^2 (estimated amount of total heterogeneity): 0.3262 (SE = 0.1471) ## tau (square root of estimated tau^2 value): 0.5712 ## I^2 (total heterogeneity / total variability): 89.40% ## H^2 (total variability / sampling variability): 9.43 ## ## Test for Heterogeneity: ## Q(df = 14) = 96.0894, p-val < .0001 ## ## Model Results: ## ## estimate se zval pval ci.lb ci.ub ## 0.8938 0.2449 3.6493 0.0003 0.4138 1.3739 *** ## ## --- ## Signif. codes: 0 '***' 0.001 '**' 0.01 '*' 0.05 '.' 0.1 ' ' 1 summary(modelref.3)

## ## Mixed-Effects Model (k = 46; tau^2 estimator: REML) ## ## logLik deviance AIC BIC AICc ## -54.6903 109.3806 115.3806 120.7332 115.9806 ## ## tau^2 (estimated amount of residual heterogeneity): 0.5135 (SE = 0.1241) ## tau (square root of estimated tau^2 value): 0.7166 ## I^2 (residual heterogeneity / unaccounted variability): 94.21% ## H^2 (unaccounted variability / sampling variability): 17.26 ## R^2 (amount of heterogeneity accounted for): 0.00% ## ## Test for Residual Heterogeneity: ## QE(df = 44) = 541.1919, p-val < .0001 ## ## Test of Moderators (coefficient(s) 2): ## QM(df = 1) = 0.0217, p-val = 0.8830 ## ## Model Results: ## ## estimate se zval pval ci.lb ci.ub ## intrcpt 0.8385 0.2188 3.8326 0.0001 0.4097 1.2674 *** ## reflect_5 0.0553 0.3755 0.1472 0.8830 -0.6807 0.7912 ## ## --- ## Signif. codes: 0 '***' 0.001 '**' 0.01 '*' 0.05 '.' 0.1 ' ' 1

Sub-Group Hands-On Learning

282

modelhol.1<-rma(measure="SMD", yi=es, vi=var, weights=wt, method="REML", subs et=(hands_6=="0"), data=Kfinal, slab=paste(id)) modelhol.2<-rma(measure="SMD", yi=es, vi=var, weights=wt, method="REML", subs et=(hands_6=="1"), data=Kfinal, slab=paste(id)) modelhol.3<-rma(measure="SMD", yi=es, vi=var, weights=wt, mods=~hands_6, meth od="REML", data=Kfinal)

summary(modelhol.1)

## ## Random-Effects Model (k = 31; tau^2 estimator: REML) ## ## logLik deviance AIC BIC AICc ## -33.9132 67.8263 71.8263 74.6287 72.2707 ## ## tau^2 (estimated amount of total heterogeneity): 0.4409 (SE = 0.1308) ## tau (square root of estimated tau^2 value): 0.6640 ## I^2 (total heterogeneity / total variability): 92.82% ## H^2 (total variability / sampling variability): 13.93 ## ## Test for Heterogeneity: ## Q(df = 30) = 309.9897, p-val < .0001 ## ## Model Results: ## ## estimate se zval pval ci.lb ci.ub ## 0.9046 0.1944 4.6526 <.0001 0.5235 1.2856 *** ## ## --- ## Signif. codes: 0 '***' 0.001 '**' 0.01 '*' 0.05 '.' 0.1 ' ' 1 summary(modelhol.2)

## ## Random-Effects Model (k = 15; tau^2 estimator: REML) ## ## logLik deviance AIC BIC AICc ## -19.9259 39.8519 43.8519 45.1300 44.9428 ## ## tau^2 (estimated amount of total heterogeneity): 0.7275 (SE = 0.3052) ## tau (square root of estimated tau^2 value): 0.8529 ## I^2 (total heterogeneity / total variability): 96.24% ## H^2 (total variability / sampling variability): 26.57 ## ## Test for Heterogeneity: ## Q(df = 14) = 225.8710, p-val < .0001 ## ## Model Results: ##

283

## estimate se zval pval ci.lb ci.ub ## 0.7749 0.3828 2.0242 0.0429 0.0246 1.5252 * ## ## --- ## Signif. codes: 0 '***' 0.001 '**' 0.01 '*' 0.05 '.' 0.1 ' ' 1 summary(modelhol.3)

## ## Mixed-Effects Model (k = 46; tau^2 estimator: REML) ## ## logLik deviance AIC BIC AICc ## -54.2743 108.5486 114.5486 119.9012 115.1486 ## ## tau^2 (estimated amount of residual heterogeneity): 0.5228 (SE = 0.1262) ## tau (square root of estimated tau^2 value): 0.7230 ## I^2 (residual heterogeneity / unaccounted variability): 94.22% ## H^2 (unaccounted variability / sampling variability): 17.31 ## R^2 (amount of heterogeneity accounted for): 0.00% ## ## Test for Residual Heterogeneity: ## QE(df = 44) = 535.8608, p-val < .0001 ## ## Test of Moderators (coefficient(s) 2): ## QM(df = 1) = 0.1118, p-val = 0.7381 ## ## Model Results: ## ## estimate se zval pval ci.lb ci.ub ## intrcpt 0.9046 0.2113 4.2818 <.0001 0.4905 1.3186 *** ## hands_6 -0.1297 0.3878 -0.3343 0.7381 -0.8898 0.6305 ## ## --- ## Signif. codes: 0 '***' 0.001 '**' 0.01 '*' 0.05 '.' 0.1 ' ' 1

Sub-Group Work Sample Analysis modelws.1<-rma(measure="SMD", yi=es, vi=var, weights=wt, method="REML", subse t=(wsan_7=="0"), data=Kfinal, slab=paste(id)) modelws.2<-rma(measure="SMD", yi=es, vi=var, weights=wt, method="REML", subse t=(wsan_7=="1"), data=Kfinal, slab=paste(id)) modelws.3<-rma(measure="SMD", yi=es, vi=var, weights=wt, mods=~wsan_7, method ="REML", data=Kfinal)

summary(modelws.1)

## ## Random-Effects Model (k = 36; tau^2 estimator: REML) ## ## logLik deviance AIC BIC AICc

284

## -38.1362 76.2723 80.2723 83.3830 80.6473 ## ## tau^2 (estimated amount of total heterogeneity): 0.3962 (SE = 0.1077) ## tau (square root of estimated tau^2 value): 0.6294 ## I^2 (total heterogeneity / total variability): 93.74% ## H^2 (total variability / sampling variability): 15.97 ## ## Test for Heterogeneity: ## Q(df = 35) = 396.5149, p-val < .0001 ## ## Model Results: ## ## estimate se zval pval ci.lb ci.ub ## 0.8437 0.1722 4.9006 <.0001 0.5063 1.1811 *** ## ## --- ## Signif. codes: 0 '***' 0.001 '**' 0.01 '*' 0.05 '.' 0.1 ' ' 1 summary(modelws.2)

## ## Random-Effects Model (k = 10; tau^2 estimator: REML) ## ## logLik deviance AIC BIC AICc ## -14.5462 29.0923 33.0923 33.4868 35.0923 ## ## tau^2 (estimated amount of total heterogeneity): 1.1739 (SE = 0.6234) ## tau (square root of estimated tau^2 value): 1.0835 ## I^2 (total heterogeneity / total variability): 94.22% ## H^2 (total variability / sampling variability): 17.31 ## ## Test for Heterogeneity: ## Q(df = 9) = 144.0375, p-val < .0001 ## ## Model Results: ## ## estimate se zval pval ci.lb ci.ub ## 0.9533 0.4315 2.2094 0.0271 0.1076 1.7990 * ## ## --- ## Signif. codes: 0 '***' 0.001 '**' 0.01 '*' 0.05 '.' 0.1 ' ' 1 summary(modelws.3)

## ## Mixed-Effects Model (k = 46; tau^2 estimator: REML) ## ## logLik deviance AIC BIC AICc ## -54.1723 108.3445 114.3445 119.6971 114.9445 ## ## tau^2 (estimated amount of residual heterogeneity): 0.5257 (SE = 0.1267)

285

## tau (square root of estimated tau^2 value): 0.7250 ## I^2 (residual heterogeneity / unaccounted variability): 94.54% ## H^2 (unaccounted variability / sampling variability): 18.30 ## R^2 (amount of heterogeneity accounted for): 0.00% ## ## Test for Residual Heterogeneity: ## QE(df = 44) = 540.5524, p-val < .0001 ## ## Test of Moderators (coefficient(s) 2): ## QM(df = 1) = 0.0952, p-val = 0.7576 ## ## Model Results: ## ## estimate se zval pval ci.lb ci.ub ## intrcpt 0.8437 0.1977 4.2667 <.0001 0.4561 1.2313 *** ## wsan_7 0.1096 0.3553 0.3086 0.7576 -0.5867 0.8059 ## ## --- ## Signif. codes: 0 '***' 0.001 '**' 0.01 '*' 0.05 '.' 0.1 ' ' 1

Sub-Group Lesson Planning modellp.1<-rma(measure="SMD", yi=es, vi=var, weights=wt, method="REML", subse t=(lps_8=="0"), data=Kfinal, slab=paste(id)) modellp.2<-rma(measure="SMD", yi=es, vi=var, weights=wt, method="REML", subse t=(lps_8=="1"), data=Kfinal, slab=paste(id)) modellp.3<-rma(measure="SMD", yi=es, vi=var, weights=wt, mods=~lps_8, method= "REML", data=Kfinal)

summary(modellp.1)

## ## Random-Effects Model (k = 19; tau^2 estimator: REML) ## ## logLik deviance AIC BIC AICc ## -23.1536 46.3071 50.3071 52.0879 51.1071 ## ## tau^2 (estimated amount of total heterogeneity): 0.6396 (SE = 0.2350) ## tau (square root of estimated tau^2 value): 0.7997 ## I^2 (total heterogeneity / total variability): 93.89% ## H^2 (total variability / sampling variability): 16.38 ## ## Test for Heterogeneity: ## Q(df = 18) = 215.3973, p-val < .0001 ## ## Model Results: ## ## estimate se zval pval ci.lb ci.ub ## 0.9947 0.2188 4.5468 <.0001 0.5659 1.4235 ***

286

## ## --- ## Signif. codes: 0 '***' 0.001 '**' 0.01 '*' 0.05 '.' 0.1 ' ' 1 summary(modellp.2)

## ## Random-Effects Model (k = 27; tau^2 estimator: REML) ## ## logLik deviance AIC BIC AICc ## -30.3554 60.7109 64.7109 67.2271 65.2326 ## ## tau^2 (estimated amount of total heterogeneity): 0.4285 (SE = 0.1383) ## tau (square root of estimated tau^2 value): 0.6546 ## I^2 (total heterogeneity / total variability): 94.20% ## H^2 (total variability / sampling variability): 17.24 ## ## Test for Heterogeneity: ## Q(df = 26) = 313.6541, p-val < .0001 ## ## Model Results: ## ## estimate se zval pval ci.lb ci.ub ## 0.7948 0.2203 3.6084 0.0003 0.3631 1.2265 *** ## ## --- ## Signif. codes: 0 '***' 0.001 '**' 0.01 '*' 0.05 '.' 0.1 ' ' 1 summary(modellp.3)

## ## Mixed-Effects Model (k = 46; tau^2 estimator: REML) ## ## logLik deviance AIC BIC AICc ## -53.7920 107.5840 113.5840 118.9366 114.1840 ## ## tau^2 (estimated amount of residual heterogeneity): 0.5166 (SE = 0.1248) ## tau (square root of estimated tau^2 value): 0.7187 ## I^2 (residual heterogeneity / unaccounted variability): 94.33% ## H^2 (unaccounted variability / sampling variability): 17.65 ## R^2 (amount of heterogeneity accounted for): 0.00% ## ## Test for Residual Heterogeneity: ## QE(df = 44) = 529.0514, p-val < .0001 ## ## Test of Moderators (coefficient(s) 2): ## QM(df = 1) = 0.4103, p-val = 0.5218 ## ## Model Results: ## ## estimate se zval pval ci.lb ci.ub

287

## intrcpt 0.9947 0.1977 5.0323 <.0001 0.6073 1.3821 *** ## lps_8 -0.1999 0.3120 -0.6405 0.5218 -0.8115 0.4117 ## ## --- ## Signif. codes: 0 '***' 0.001 '**' 0.01 '*' 0.05 '.' 0.1 ' ' 1

Sub-Group Study quality modqs.1<-rma(yi=es, vi=var, weights=wt, method="REML", subset=(qs_rank=="low" ), data=Kfinal, slab=paste(id)) modqs.2<-rma(yi=es, vi=var, weights=wt, method="REML", subset=(qs_rank=="ave" ), data=Kfinal, slab=paste(id)) modqs.3<-rma(yi=es, vi=var, weights=wt, method="REML", subset=(qs_rank=="high "), data=Kfinal, slab=paste(id)) modqs.4<-rma(yi=es, vi=var, weights=wt, mods=~qs_rank, method="REML", data=Kf inal)

summary(modqs.1)

## ## Random-Effects Model (k = 9; tau^2 estimator: REML) ## ## logLik deviance AIC BIC AICc ## -11.0353 22.0707 26.0707 26.2296 28.4707 ## ## tau^2 (estimated amount of total heterogeneity): 0.8188 (SE = 0.4489) ## tau (square root of estimated tau^2 value): 0.9049 ## I^2 (total heterogeneity / total variability): 93.87% ## H^2 (total variability / sampling variability): 16.31 ## ## Test for Heterogeneity: ## Q(df = 8) = 131.3886, p-val < .0001 ## ## Model Results: ## ## estimate se zval pval ci.lb ci.ub ## 1.1759 0.3681 3.1941 0.0014 0.4543 1.8974 ** ## ## --- ## Signif. codes: 0 '***' 0.001 '**' 0.01 '*' 0.05 '.' 0.1 ' ' 1 summary(modqs.2)

## ## Random-Effects Model (k = 26; tau^2 estimator: REML) ## ## logLik deviance AIC BIC AICc ## -24.7772 49.5543 53.5543 55.9921 54.0998 ## ## tau^2 (estimated amount of total heterogeneity): 0.2815 (SE = 0.0941)

288

## tau (square root of estimated tau^2 value): 0.5305 ## I^2 (total heterogeneity / total variability): 92.07% ## H^2 (total variability / sampling variability): 12.61 ## ## Test for Heterogeneity: ## Q(df = 25) = 228.2873, p-val < .0001 ## ## Model Results: ## ## estimate se zval pval ci.lb ci.ub ## 0.8371 0.1726 4.8504 <.0001 0.4989 1.1754 *** ## ## --- ## Signif. codes: 0 '***' 0.001 '**' 0.01 '*' 0.05 '.' 0.1 ' ' 1 summary(modqs.3)

## ## Random-Effects Model (k = 11; tau^2 estimator: REML) ## ## logLik deviance AIC BIC AICc ## -16.2505 32.5011 36.5011 37.1062 38.2154 ## ## tau^2 (estimated amount of total heterogeneity): 1.0623 (SE = 0.5273) ## tau (square root of estimated tau^2 value): 1.0307 ## I^2 (total heterogeneity / total variability): 95.54% ## H^2 (total variability / sampling variability): 22.42 ## ## Test for Heterogeneity: ## Q(df = 10) = 157.9399, p-val < .0001 ## ## Model Results: ## ## estimate se zval pval ci.lb ci.ub ## 0.7012 0.4498 1.5590 0.1190 -0.1804 1.5827 ## ## --- ## Signif. codes: 0 '***' 0.001 '**' 0.01 '*' 0.05 '.' 0.1 ' ' 1 summary(modqs.4)

## ## Mixed-Effects Model (k = 46; tau^2 estimator: REML) ## ## logLik deviance AIC BIC AICc ## -54.5947 109.1893 117.1893 124.2341 118.2420 ## ## tau^2 (estimated amount of residual heterogeneity): 0.5403 (SE = 0.1315) ## tau (square root of estimated tau^2 value): 0.7350 ## I^2 (residual heterogeneity / unaccounted variability): 94.56% ## H^2 (unaccounted variability / sampling variability): 18.37

289

## R^2 (amount of heterogeneity accounted for): 0.00% ## ## Test for Residual Heterogeneity: ## QE(df = 43) = 517.6158, p-val < .0001 ## ## Test of Moderators (coefficient(s) 2:3): ## QM(df = 2) = 1.2803, p-val = 0.5272 ## ## Model Results: ## ## estimate se zval pval ci.lb ci.ub ## intrcpt 0.8371 0.2374 3.5257 0.0004 0.3718 1.3025 *** ## qs_rankhigh -0.1359 0.4016 -0.3385 0.7350 -0.9230 0.6511 ## qs_ranklow 0.3388 0.3843 0.8814 0.3781 -0.4145 1.0921 ## ## --- ## Signif. codes: 0 '***' 0.001 '**' 0.01 '*' 0.05 '.' 0.1 ' ' 1

Sub-Group Validity modelv.1<-rma(yi=es, vi=var, weights=wt, method="REML", subset=(valid=="SST") , data=Kfinal, slab=paste(id)) modelv.2<-rma(yi=es, vi=var, weights=wt, method="REML", subset=(valid=="PVM") , data=Kfinal, slab=paste(id)) modelv.3<-rma(yi=es, vi=var, weights=wt, method="REML", subset=(valid=="PVWOM "), data=Kfinal, slab=paste(id)) modelv.4<-rma(yi=es, vi=var, weights=wt, mods=~valid, method="REML", data=Kfi nal)

summary(modelv.1)

## ## Random-Effects Model (k = 11; tau^2 estimator: REML) ## ## logLik deviance AIC BIC AICc ## -7.5425 15.0850 19.0850 19.6902 20.7993 ## ## tau^2 (estimated amount of total heterogeneity): 0.1828 (SE = 0.0941) ## tau (square root of estimated tau^2 value): 0.4276 ## I^2 (total heterogeneity / total variability): 92.92% ## H^2 (total variability / sampling variability): 14.12 ## ## Test for Heterogeneity: ## Q(df = 10) = 98.3145, p-val < .0001 ## ## Model Results: ## ## estimate se zval pval ci.lb ci.ub ## 0.7523 0.1804 4.1698 <.0001 0.3987 1.1059 ***

290

## ## --- ## Signif. codes: 0 '***' 0.001 '**' 0.01 '*' 0.05 '.' 0.1 ' ' 1 summary(modelv.2)

## ## Random-Effects Model (k = 16; tau^2 estimator: REML) ## ## logLik deviance AIC BIC AICc ## -19.4822 38.9644 42.9644 44.3805 43.9644 ## ## tau^2 (estimated amount of total heterogeneity): 0.7350 (SE = 0.2892) ## tau (square root of estimated tau^2 value): 0.8573 ## I^2 (total heterogeneity / total variability): 94.90% ## H^2 (total variability / sampling variability): 19.62 ## ## Test for Heterogeneity: ## Q(df = 15) = 291.5185, p-val < .0001 ## ## Model Results: ## ## estimate se zval pval ci.lb ci.ub ## 1.1221 0.2709 4.1427 <.0001 0.5912 1.6530 *** ## ## --- ## Signif. codes: 0 '***' 0.001 '**' 0.01 '*' 0.05 '.' 0.1 ' ' 1 summary(modelv.3)

## ## Random-Effects Model (k = 19; tau^2 estimator: REML) ## ## logLik deviance AIC BIC AICc ## -24.9835 49.9671 53.9671 55.7478 54.7671 ## ## tau^2 (estimated amount of total heterogeneity): 0.5859 (SE = 0.2322) ## tau (square root of estimated tau^2 value): 0.7654 ## I^2 (total heterogeneity / total variability): 89.66% ## H^2 (total variability / sampling variability): 9.67 ## ## Test for Heterogeneity: ## Q(df = 18) = 110.9966, p-val < .0001 ## ## Model Results: ## ## estimate se zval pval ci.lb ci.ub ## 0.7680 0.2183 3.5188 0.0004 0.3402 1.1958 *** ## ## --- ## Signif. codes: 0 '***' 0.001 '**' 0.01 '*' 0.05 '.' 0.1 ' ' 1

291

summary(modelv.4)

## ## Mixed-Effects Model (k = 46; tau^2 estimator: REML) ## ## logLik deviance AIC BIC AICc ## -53.6521 107.3042 115.3042 122.3490 116.3569 ## ## tau^2 (estimated amount of residual heterogeneity): 0.5341 (SE = 0.1302) ## tau (square root of estimated tau^2 value): 0.7308 ## I^2 (residual heterogeneity / unaccounted variability): 94.43% ## H^2 (unaccounted variability / sampling variability): 17.94 ## R^2 (amount of heterogeneity accounted for): 0.00% ## ## Test for Residual Heterogeneity: ## QE(df = 43) = 500.8296, p-val < .0001 ## ## Test of Moderators (coefficient(s) 2:3): ## QM(df = 2) = 1.5429, p-val = 0.4623 ## ## Model Results: ## ## estimate se zval pval ci.lb ci.ub ## intrcpt 1.1221 0.2323 4.8307 <.0001 0.6668 1.5774 *** ## validPVWOM -0.3541 0.3126 -1.1329 0.2573 -0.9667 0.2585 ## validSST -0.3698 0.3832 -0.9652 0.3345 -1.1208 0.3812 ## ## --- ## Signif. codes: 0 '***' 0.001 '**' 0.01 '*' 0.05 '.' 0.1 ' ' 1

Sub-Group Reliability modelrel.1<-rma(measure="SMD", yi=es, vi=var, weights=wt, method="REML", subs et=(reliable=="0"), data=Kfinal, slab=paste(id)) modelrel.2<-rma(measure="SMD", yi=es, vi=var, weights=wt, method="REML", subs et=(reliable=="1"), data=Kfinal, slab=paste(id)) modelrel.3<-rma(measure="SMD", yi=es, vi=var, weights=wt, mods=~reliable, met hod="REML", data=Kfinal)

summary(modelrel.1)

## ## Random-Effects Model (k = 23; tau^2 estimator: REML) ## ## logLik deviance AIC BIC AICc ## -23.0806 46.1612 50.1612 52.3433 50.7928 ## ## tau^2 (estimated amount of total heterogeneity): 0.3254 (SE = 0.1198) ## tau (square root of estimated tau^2 value): 0.5704 ## I^2 (total heterogeneity / total variability): 87.69%

292

## H^2 (total variability / sampling variability): 8.12 ## ## Test for Heterogeneity: ## Q(df = 22) = 192.4665, p-val < .0001 ## ## Model Results: ## ## estimate se zval pval ci.lb ci.ub ## 0.9697 0.1585 6.1177 <.0001 0.6590 1.2803 *** ## ## --- ## Signif. codes: 0 '***' 0.001 '**' 0.01 '*' 0.05 '.' 0.1 ' ' 1 summary(modelrel.2)

## ## Random-Effects Model (k = 23; tau^2 estimator: REML) ## ## logLik deviance AIC BIC AICc ## -29.6604 59.3208 63.3208 65.5029 63.9524 ## ## tau^2 (estimated amount of total heterogeneity): 0.7139 (SE = 0.2342) ## tau (square root of estimated tau^2 value): 0.8450 ## I^2 (total heterogeneity / total variability): 96.79% ## H^2 (total variability / sampling variability): 31.19 ## ## Test for Heterogeneity: ## Q(df = 22) = 339.3440, p-val < .0001 ## ## Model Results: ## ## estimate se zval pval ci.lb ci.ub ## 0.7972 0.2933 2.7177 0.0066 0.2223 1.3721 ** ## ## --- ## Signif. codes: 0 '***' 0.001 '**' 0.01 '*' 0.05 '.' 0.1 ' ' 1 summary(modelrel.3)

## ## Mixed-Effects Model (k = 46; tau^2 estimator: REML) ## ## logLik deviance AIC BIC AICc ## -54.0962 108.1923 114.1923 119.5449 114.7923 ## ## tau^2 (estimated amount of residual heterogeneity): 0.5250 (SE = 0.1267) ## tau (square root of estimated tau^2 value): 0.7245 ## I^2 (residual heterogeneity / unaccounted variability): 94.40% ## H^2 (unaccounted variability / sampling variability): 17.85 ## R^2 (amount of heterogeneity accounted for): 0.00% ##

293

## Test for Residual Heterogeneity: ## QE(df = 44) = 531.8105, p-val < .0001 ## ## Test of Moderators (coefficient(s) 2): ## QM(df = 1) = 0.2892, p-val = 0.5907 ## ## Model Results: ## ## estimate se zval pval ci.lb ci.ub ## intrcpt 0.9697 0.1984 4.8882 <.0001 0.5809 1.3584 *** ## reliable -0.1725 0.3207 -0.5378 0.5907 -0.8011 0.4561 ## ## --- ## Signif. codes: 0 '***' 0.001 '**' 0.01 '*' 0.05 '.' 0.1 ' ' 1

294

APPENDIX N ATTITUDE STUDIES

The 28 studies listed here were used as the data sources for the analyses related to attitude. Both Abbitt and Klett (2007) and Ward and Overall (2011) contributed four populations each, leading to the overall 34 studies analyzed for the attitude results.

Abbitt, J. T. (2011). An investigation of the relationship between self-efficacy beliefs about technology integration and technological pedagogical content knowledge (TPACK) among preservice teachers. Journal of Digital Learning in Teacher Education, 27(4), 134-143.

Abbitt, J. T., & Klett, M. D. (2007). Identifying influences on attitudes and self-efficacy beliefs towards technology integration among preservice educators. Electronic Journal for the integration of technology in Education, 6(1), 28-42.

Agyei, D. D., & Voogt, J. M. (2015). Preservice teachers’ TPACK competencies for spreadsheet integration: insights from a mathematics-specific instructional technology course. Technology, Pedagogy and Education, 24(5), 605-625.

Akkaya, R. (2016). Research on the Development of Middle School Mathematics Preservice Teachers' Perceptions Regarding the Use of Technology in Teaching Mathematics. Eurasia Journal of Mathematics, Science & Technology Education, 12(4), 861-879.

Alayyar, G. M., Fisser, P., & Voogt, J. (2012). Developing technological pedagogical content knowledge in preservice science teachers: Support from blended learning. Australasian Journal of Educational Technology, 28(8).

Alexander, C., Knezek, G., Christensen, R., Tyler-Wood, T., & Bull, G. (2014). The impact of project-based learning on preservice teachers’ technology attitudes and skills. Journal of Computers in Mathematics and Science Teaching, 33(3), 257-282.

An, H., Wilder, H., & Lim, K. (2011). Preparing Elementary Preservice Teachers from a Non- Traditional Student Population to Teach with Technology. Computers in the Schools, 28(2), 170- 193.

Anderson, S. E., & Maninger, R. M. (2007). Preservice teachers' abilities, beliefs, and intentions regarding technology integration. Journal of Educational Computing Research, 37(2), 151-172.

Bai, H., & Ertmer, P. A. (2008). Teacher educators' beliefs and technology uses as predictors of preservice teachers' beliefs and technology attitudes. Journal of Technology and Teacher Education, 16(1), 93.

Browne, J. M. (2007). Evidence supporting the validity of inferences required by the intended uses of the Technology Integration Confidence Scale.

295

Cengiz, C. (2015). The development of TPACK, Technology Integrated Self-Efficacy and Instructional Technology Outcome Expectations of preservice physical education teachers. Asia- Pacific Journal of Teacher Education, 43(5), 411-422.

Clark, C., Zhang, S., & Strudler, N. (2015). Teacher candidate technology integration: For student learning or instruction? Journal of Digital Learning in Teacher Education, 31(3), 93- 106.

Duran, M., & Fossum, P. R. (2010). Technology integration into teacher preparation: Part 1 - current practice and theoretical grounding for pedagogical renewal. Ahi Evran Üniversitesi Kırşehir Eğitim Fakültesi Dergisi, 11(2).

Heo, M. (2009). Digital storytelling: An empirical study of the impact of digital storytelling on preservice teachers' self-efficacy and dispositions towards educational technology. Journal of Educational Multimedia and Hypermedia, 18(4), 405.

Heo, M. (2011). Improving technology competency and disposition of beginning preservice teachers with digital storytelling. Journal of Educational Multimedia and Hypermedia, 20(1), 61- 81.

Hur, S. J. (2009). Preparing Preservice Teachers for Technology Integration with Multimedia Resources. Paper presented at the EdMedia: World Conference on Educational Media and Technology.

Jang, S.-J. (2008). The effects of integrating technology, observation and writing into a teacher education method course. Computers & Education, 50(3), 853-865.

Johnson, L. D. (2012). The effect of design teams on preservice teachers' technology integration. Syracuse University.

Koh, J. H., & Frick, T. W. (2009). Instructor and student classroom interactions during technology skills instruction for facilitating preservice teachers' computer self-efficacy. Journal of Educational Computing Research, 40(2), 211-228.

Kounenou, K., Roussos, P., Yotsidi, V., & Tountopoulou, M. (2015). Trainee Teachers’ Intention to Incorporating ICT Use into Teaching Practice in Relation to their Psychological Characteristics: The Case of Group-based Intervention. Procedia-Social and Behavioral Sciences, 190, 120-128.

Lambert, J., Gong, Y., & Cuper, P. (2008). Technology, transfer, and teaching: The impact of a single technology course on preservice teachers' computer attitudes and ability. Journal of Technology and Teacher Education, 16(4), 385.

Lee, Y., & Lee, J. (2014). Enhancing preservice teachers' self-efficacy beliefs for technology integration through lesson planning practice. Computers & Education, 73, 121-128.

296

Milman, N. B., & Molebash, P. E. (2008). A longitudinal assessment of teacher education students' confidence toward using technology. Journal of Educational Computing Research, 38(2), 183-200.

Mizell, S. (2016). Factors affecting early adoption of technology.

Papastergiou, M. (2010). Enhancing physical education and sport science students’ self-efficacy and attitudes regarding information and communication technologies through a computer literacy course. Computers & Education, 54(1), 298-308.

Southall, S. P. (2012). Digital native preservice teachers: An examination of their self-efficacy beliefs regarding technology integration in classroom settings: Virginia Commonwealth University.

Ward, G., & Overall, T. (2011). Technology integration for preservice teachers: Evaluating the team-taught cohort model. Journal of Technology and Teacher Education, 19(1), 23-43.

Willis, J. M. (2015). Examining technology and teaching efficacy of preservice teacher candidates: A deliberate course design model. Current Issues in Education, 18(3).

297

APPENDIX O BELIEFS STUDIES

The 12 studies listed here were used as the data sources for the analyses related to belief.

Anderson, S. E., & Maninger, R. M. (2007). Preservice teachers' abilities, beliefs, and intentions regarding technology integration. Journal of Educational Computing Research, 37(2), 151-172.

Clark, C., Zhang, S., & Strudler, N. (2015). Teacher candidate technology integration: For student learning or instruction? Journal of Digital Learning in Teacher Education, 31(3), 93- 106.

Jang, S.-J. (2008). The effects of integrating technology, observation and writing into a teacher education method course. Computers & Education, 50(3), 853-865.

Karatas, I. (2014). Changing preservice mathematics teachers’ beliefs about using computers for teaching and learning mathematics: the effect of three different models. European Journal of Teacher Education, 37(3), 390-405.

Lim, C. P., & Chan, B. C. (2007). MicroLESSONS in teacher education: Examining preservice teachers’ pedagogical beliefs. Computers & Education, 48(3), 474-494.

Mizell, S. (2016). Factors affecting early adoption of technology.

Willis, J. M. (2015). Examining technology and teaching efficacy of preservice teacher candidates: A deliberate course design model. Current Issues in Education, 18(3).

298

APPENDIX P KNOWLEDGE STUDIES

The 38 studies listed here were used as the data sources for the analyses related to knowledge.

Pheeraphan (2013) and Starčič, Cotic, Solomonides, and Volk (2016), and Mitchell (2013) each contributed two populations to the data. Kramarski and Michalsky (2009) contributed three populations. Ward and Overall (2011) contributed four populations. This lead to a final analysis of 46 studies.

Agyei, D. D., & Keengwe, J. (2014). Using technology pedagogical content knowledge development to enhance learning outcomes. Education and Information technologies, 19(1), 155-171.

An, H., Wilder, H., & Lim, K. (2011). Preparing Elementary Preservice Teachers from a Non- Traditional Student Population to Teach with Technology. Computers in the Schools, 28(2), 170- 193.

Anderson, S. E., & Maninger, R. M. (2007). Preservice teachers' abilities, beliefs, and intentions regarding technology integration. Journal of Educational Computing Research, 37(2), 151-172.

299

Blankson, J., Keengwe, J., & Kyei-Blankson, L. (2010). Teachers and technology: Enhancing technology competencies for preservice teachers. International Journal of Information and Communication Technology Education (IJICTE), 6(1), 45-54.

Chai, C. S., Koh, J. H. L., & Tsai, C.-C. (2010). Facilitating Preservice Teachers' Development of Technological, Pedagogical, and Content Knowledge (TPACK). Educational Technology & Society, 13(4), 63-73.

Chai, C. S., Ling Koh, J. H., Tsai, C.-C., & Lee Wee Tan, L. (2011). Modeling primary school preservice teachers’ Technological Pedagogical Content Knowledge (TPACK) for meaningful learning with information and communication technology (ICT). Computers & Education, 57(1), 1184-1193.

Coffman, V. G. (2013). The perceived technology proficiency of students in a teacher education program: Kent State University.

Ersoy, M., Yurdakul, I. K., & Ceylan, B. (2016). Investigating preservice teachers' TPACK competencies through the lenses of ICT skills: An experimental study. Egitim ve Bilim, 41(186).

Goodwin, A. H. (2012). Building preservice teachers' knowledge of content, pedagogy, and technology through TPACK-infused course experiences: A study replication and extension: University of Kentucky.

Graham, C. R., Borup, J., & Smith, N. B. (2012). Using TPACK as a framework to understand teacher candidates' technology integration decisions. Journal of Computer Assisted Learning, 28(6), 530-546.

Han, I., Eom, M., & Shin, W. S. (2013). Multimedia case-based learning to enhance preservice teachers' knowledge integration for teaching with technologies. Teaching and Teacher Education, 34, 122-129.

Hur, S. J. (2009). Preparing Preservice Teachers for Technology Integration with Multimedia Resources. Paper presented at the EdMedia: World Conference on Educational Media and Technology.

Johnson, L. D. (2012). The effect of design teams on preservice teachers' technology integration. Syracuse University.

300

Kafyulilo, A., Fisser, P., Pieters, J., & Voogt, J. (2015). ICT use in science and mathematics teacher education in Tanzan: Developing Technological Pedagogical Content Knowledge. Australasian Journal of Educational Technology, 31(4), 381-399.

Kohen, Z., & Kramarski, B. (2012). Developing a TPCK-SRL assessment scheme for conceptually advancing technology in education. Studies in Educational Evaluation, 38(1), 1-8.

Kramarski, B., & Michalsky, T. (2009). Three metacognitive approaches to training preservice teachers in different learning phases of technological pedagogical content knowledge. Educational Research and Evaluation, 15(5), 465-485.

Lee, C., & Kim, C. (2014). An implementation study of a TPACK-based instructional design model in a technology integration course. Educational Technology Research and Development, 62(4), 437-460.

Lyublinskaya, I., & Tournaki, N. (2014). A Study of special education teachers’ TPACK development in mathematics and science through assessment of lesson plans. Journal of Technology and Teacher Education, 22(4), 449-470.

Martinovic, D., & Zhang, Z. (2012). Situating ICT in the teacher education program: Overcoming challenges, fulfilling expectations. Teaching and Teacher Education, 28(3), 461- 469.

Mettas, A. C., & Constantinou, C. C. (2008). The technology fair: a project-based learning approach for enhancing problem solving skills and interest in design and technology education. International Journal of Technology and Design Education, 18(1), 79-100.

Mitchell, M. (2013). A comparative study of mathematical content, pedagogy and technology using a paired t-test of the means of seven domains. Paper presented at the Society for Information Technology & Teacher Education International Conference.

Mizell, S. (2016). Factors affecting early adoption of technology.

Mouza, C., Karchmer-Klein, R., Nandakumar, R., Ozden, S. Y., & Hu, L. (2014). Investigating the impact of an integrated approach to the development of preservice teachers' technological pedagogical content knowledge (TPACK). Computers & Education, 71, 206-221.

Pheeraphan, N. (2013). Enhancement of the 21st century skills for Thai higher education by integration of ICT in classroom. Procedia-Social and Behavioral Sciences, 103, 365-373.

Sabo, K. (2013). A Mixed-Methods Examination of Influences on the Shape and Malleability of Technological Pedagogical Content Knowledge (TPACK) in Graduate Teacher Education Students: Arizona State University.

301

Schmidt, D., Baran, E., Thompson, A., Koehler, M., Punya, M., & Shin, T. (2009). Examining preservice teachers' development of technological pedagogical content knowledge in an introductory instructional technology course. Paper presented at the Society for Information Technology & Teacher Education International Conference.

Shah, M. (2015). Preservice teacher education in game-based learning: Cultivating knowledge and skills for integrating digital games in K-12 classrooms: Drexel University.

Shinas, V. H., Karchmer-Klein, R., Mouza, C., Yilmaz-Ozden, S., & J. Glutting, J. (2015). Analyzing preservice teachers' Technological Pedagogical Content Knowledge development in the context of a multidimensional teacher preparation program. Journal of Digital Learning in Teacher Education, 31(2), 47-55.

Southall, S. P. (2012). Digital native preservice teachers: An examination of their self-efficacy beliefs regarding technology integration in classroom settings: Virginia Commonwealth University.

Starčič, A. I., Cotic, M., Solomonides, I., & Volk, M. (2016). Engaging preservice primary and preprimary school teachers in digital storytelling for the teaching and learning of mathematics. British journal of educational technology, 47(1), 29-50.

Ward, G., & Overall, T. (2011). Technology integration for preservice teachers: Evaluating the team-taught cohort model. Journal of Technology and Teacher Education, 19(1), 23-43.

302

APPENDIX Q ATTITUDE RESEARCH DESIGN DATA

Does the Was the Was the Was the research need for the research research clearly define research grounded in clearly the Authors well stated? theory? described? population? Kounenou et al. (2015) yes yes yes yes Browne (2007) yes yes yes no Alexander et al. (2014) yes yes yes yes Alayyar et al. (2012) yes yes yes yes Cengiz (2015) yes yes yes yes Y. Lee and Lee (2014) yes yes yes yes Abbitt and Klett (2007) yes yes yes yes Milman and Molebash (2008) yes yes yes yes Anderson and Maninger (2007) yes yes yes yes Bai and Ertmer (2008) yes yes yes yes Willis (2015) yes yes yes yes Lambert et al. (2008) yes yes yes yes Jang (2008) yes yes yes yes Clark et al. (2015) yes yes yes yes Ward and Overall (2011) yes yes yes no Mizell (2016) yes yes yes yes Agyei and Voogt (2015) yes yes yes yes Heo (2009) yes yes yes no An et al. (2011) yes yes yes yes Heo (2011) yes yes yes yes Akkaya (2016) yes yes yes no Koh and Frick (2009) yes yes yes no Hur (2009) yes no yes no Duran and Fossum (2010) yes yes yes yes Abbitt (2011) yes yes yes yes Southall (2012) yes yes yes yes Papastergiou (2010) yes yes yes no Johnson (2012) yes yes yes yes

303

Did the study Did the study report any address sub-group potential bias in Did the study analyses (i.e., how the course report on student gender, under participants that ethnicity, examination declined Authors etc.)? was selected? participation? Kounenou et al. (2015) yes no no Browne (2007) yes no yes Alexander et al. (2014) yes no no Alayyar et al.(2012) yes no yes Cengiz (2015) no yes yes Y. Lee and Lee (2014) yes no no Abbitt and Klett (2007) no no no Milman and Molebash (2008) no no no Anderson and Maninger (2007) yes yes no Bai and Ertmer (2008) yes no yes Willis (2015) no no yes Lambert et al. (2008) yes no no Jang (2008) yes no yes Clark et al. (2015) no no no Ward and Overall (2011) no yes yes Mizell (2016) no yes yes Agyei and Voogt (2015) no no no Heo (2009) yes no yes An et al. (2011) no no yes Heo (2011) no yes yes Akkaya (2016) no no no Koh and Frick (2009) no yes yes Hur (2009) no no no Duran and Fossum (2010) yes yes yes Abbitt (2011) no no yes Southall (2012) yes yes yes Papastergiou (2010) yes no no Johnson (2012) yes yes yes

304

Did the study authors report on the appropriateness of What kind of sample size for impact on sampling was statistical quality and/or used for the Authors conduct analyses for power? study? Kounenou et al. (2015) not appropriate/reported non-probability Browne (2007) not appropriate/reported non-probability Alexander et al. (2014) not appropriate/reported non-probability Alayyar et al.(2012) not appropriate/reported non-probability Cengiz (2015) not appropriate/reported non-probability Y. Lee and Lee (2014) not appropriate/reported non-probability Abbitt and Klett (2007) not appropriate/reported non-probability Milman and Molebash (2008) not appropriate/reported non-probability Anderson and Maninger (2007) not appropriate/reported non-probability Bai and Ertmer (2008) not appropriate/reported non-probability Willis (2015) not appropriate/reported non-probability Lambert et al. (2008) not appropriate/reported non-probability Jang (2008) not appropriate/reported non-probability Clark et al. (2015) not appropriate/reported non-probability Ward and Overall (2011) not appropriate/reported non-probability Mizell (2016) not appropriate/reported non-probability Agyei and Voogt (2015) not appropriate/reported non-probability Heo (2009) power analysis non-probability An et al. (2011) not appropriate/reported non-probability Heo (2011) power analysis non-probability Akkaya (2016) not appropriate/reported non-probability Koh and Frick (2009) not appropriate/reported non-probability Hur (2009) not appropriate/reported non-probability Duran and Fossum (2010) not appropriate/reported probability Abbitt (2011) not appropriate/reported non-probability Southall (2012) power analysis non-probability Papastergiou (2010) not appropriate/reported non-probability Johnson (2012) not appropriate/reported non-probability

305

Did the researchers Was there any randomize management participants, of pre-course instructor, Was a comparison differences assessor group used and between measuring appropriate for the comparison outcome, and Authors study? groups? data analyst? Kounenou et al. (2015) no comparison group no single Browne (2007) no comparison group yes no/unknown Alexander et al. (2014) no comparison group no no/unknown Alayyar et al.(2012) appropriate no no/unknown Cengiz (2015) no comparison group no no/unknown Y. Lee and Lee (2014) no comparison group yes no/unknown Abbitt and Klett (2007) no comparison group no no/unknown Milman and Molebash (2008) no comparison group no no/unknown Anderson and Maninger (2007) no comparison group no no/unknown Bai and Ertmer (2008) appropriate yes single Willis (2015) no comparison group no no/unknown Lambert et al. (2008) no comparison group yes no/unknown Jang (2008) appropriate yes no/unknown Clark et al. (2015) no comparison group no no/unknown Ward and Overall (2011) appropriate no no/unknown Mizell (2016) no comparison group no no/unknown Agyei and Voogt (2015) appropriate no no/unknown Heo (2009) no comparison group yes no/unknown An et al. (2011) no comparison group no no/unknown Heo (2011) no comparison group no no/unknown Akkaya (2016) no comparison group no no/unknown Koh and Frick (2009) no comparison group no no/unknown Hur (2009) no comparison group no no/unknown Duran and Fossum (2010) no comparison group yes multiple Abbitt (2011) no comparison group no no/unknown Southall (2012) no comparison group yes no/unknown Papastergiou (2010) no comparison group no no/unknown Johnson (2012) appropriate yes single

306

To what extent was Did the Would the the course Was the study define course be design integrity of technologies replicable consistent the course related to Authors by others? with theory? maintained? the course? Kounenou et al. (2015) some high no/unknown no Browne (2007) some not/unknown no/unknown no Alexander et al. (2014) some not/unknown yes vague Alayyar et al.(2012) some not/unknown maybe no Cengiz (2015) some not/unknown no/unknown vague Y. Lee and Lee (2014) some high yes vague Abbitt and Klett (2007) some low yes vague Milman and Molebash (2008) some high yes vague Anderson and Maninger (2007) some not/unknown no/unknown vague Bai and Ertmer (2008) some not/unknown no/unknown no Willis (2015) some high yes vague Lambert et al. (2008) highly high yes detailed Jang (2008) highly high yes no Clark et al. (2015) some high no/unknown vague Ward and Overall (2011) some not/unknown yes no Mizell (2016) some not/unknown no/unknown no Agyei and Voogt (2015) some high yes vague Heo (2009) some not/unknown no/unknown vague An et al. (2011) some not/unknown no/unknown no Heo (2011) some low yes vague Akkaya (2016) highly high yes detailed Koh and Frick (2009) some not/unknown no/unknown no Hur (2009) some low maybe vague Duran and Fossum (2010) some high yes vague Abbitt (2011) some not/unknown no/unknown no Southall (2012) some not/unknown no/unknown no Papastergiou (2010) highly high yes detailed Johnson (2012) highly high yes detailed

307

Was study attrition prevented and/or What was the measurement Authors reported? period between measures? Kounenou et al. (2015) not reported unknown Browne (2007) not reported beginning/end semester Alexander et al. (2014) not reported one class Alayyar et al.(2012) not reported beginning/end semester Cengiz (2015) not reported beginning/end semester Y. Lee and Lee (2014) not reported beginning/end semester Abbitt and Klett (2007) not reported beginning/end semester Milman and Molebash (2008) not reported beginning/end semester Anderson and Maninger (2007) not reported beginning/end semester Bai and Ertmer (2008) not reported beginning/end semester Willis (2015) not reported beginning/end semester Lambert et al. (2008) not reported beginning/end semester Jang (2008) not reported beginning/end semester Clark et al. (2015) not reported beginning/end semester Ward and Overall (2011) not reported beginning/end semester Mizell (2016) not reported beginning/end semester Agyei and Voogt (2015) not reported multiple weeks Heo (2009) not reported one class An et al. (2011) not prevented beginning/end semester Heo (2011) not prevented one class Akkaya (2016) not reported beginning/end semester Koh and Frick (2009) not reported beginning/end semester Hur (2009) not reported beginning/end semester Duran and Fossum (2010) not reported beginning/end semester Abbitt (2011) not reported beginning/end semester Southall (2012) not prevented beginning/end semester Papastergiou (2010) not reported beginning/end semester Johnson (2012) not reported beginning/end semester

308

Did the researchers Were establish confounders Were exact assumptions of controlled in test statistic analysis design values and p consistent with addressed in levels Authors data? analysis? presented? Kounenou et al. (2015) no no no Browne (2007) consistent yes no Alexander et al. (2014) no no no Alayyar et al.(2012) no no yes Cengiz (2015) no no no Y. Lee and Lee (2014) consistent yes no Abbitt and Klett (2007) no no no Milman and Molebash (2008) adjusted no no Anderson and Maninger (2007) no no no Bai and Ertmer (2008) no yes no Willis (2015) no no no Lambert et al. (2008) consistent no no Jang (2008) no no yes Clark et al. (2015) adjusted no no Ward and Overall (2011) no no yes Mizell (2016) no no no Agyei and Voogt (2015) no no no Heo (2009) no no yes An et al. (2011) consistent no no Heo (2011) no no yes Akkaya (2016) consistent no yes Koh and Frick (2009) no no no Hur (2009) no no yes Duran and Fossum (2010) no no yes Abbitt (2011) no no no Southall (2012) adjusted yes yes Papastergiou (2010) no no no Johnson (2012) consistent yes no

309

APPENDIX R BELIEFS RESEARCH DESIGN DATA

Was the need for Was the the Was the Was the research research research research clearly well grounded in clearly define the Authors stated? theory? described? population? Kounenou et al. (2015) yes yes yes yes Lim and Chan (2007) yes yes yes no Alayyar et al. (2012) yes yes yes yes Cengiz (2015) yes yes yes yes Anderson and Maninger (2007) yes yes yes yes Bai and Ertmer (2008) yes yes yes yes Willis (2015) yes yes yes yes Lambert et al. (2008) yes yes yes yes Jang (2008) yes yes yes yes Clark et al. (2015) yes yes yes yes Mizell (2016) yes yes yes yes Karatas (2014) yes yes yes no

310

311

Did the study authors report on the appropriateness of sample size for impact on What kind of statistical quality and/or sampling was conduct analyses for used for the Authors power? study? Kounenou et al. (2015) not appropriate/reported non-probability Lim and Chan (2007) not appropriate/reported non-probability Alayyar et al. (2012) not appropriate/reported non-probability Cengiz (2015) not appropriate/reported non-probability Anderson and Maninger (2007) not appropriate/reported non-probability Bai and Ertmer (2008) not appropriate/reported non-probability Willis (2015) not appropriate/reported non-probability Lambert et al. (2008) not appropriate/reported non-probability Jang (2008) not appropriate/reported non-probability Clark et al. (2015) not appropriate/reported non-probability Mizell (2016) not appropriate/reported non-probability Karatas (2014) not appropriate/reported non-probability

312

313

To what extent Would the was the course Was the course be design integrity of the replicable by consistent with course Authors others? theory? maintained? Kounenou et al. (2015) some high no/unknown Lim and Chan (2007) highly high yes Alayyar et al. (2012) some not/unknown maybe Cengiz (2015) some not/unknown no/unknown Anderson and Maninger (2007) some not/unknown no/unknown Bai and Ertmer (2008) some not/unknown no/unknown Willis (2015) some high yes Lambert et al. (2008) highly high yes Jang (2008) highly high yes Clark et al. (2015) some high no/unknown Mizell (2016) some not/unknown no/unknown Karatas (2014) some high yes

314

Did the study Was study define attrition technologies prevented What was the related to the and/or measurement period Authors course? reported? between measures? Kounenou et al. (2015) no not reported unknown Lim and Chan (2007) vague not reported multiple weeks Alayyar et al. (2012) no not reported beginning/end semester Cengiz (2015) vague not reported beginning/end semester Anderson and Maninger (2007) vague not reported beginning/end semester Bai and Ertmer (2008) no not reported beginning/end semester Willis (2015) vague not reported beginning/end semester Lambert et al. (2008) detailed not reported beginning/end semester Jang (2008) no not reported beginning/end semester Clark et al. (2015) vague not reported beginning/end semester Mizell (2016) no not reported beginning/end semester Karatas (2014) vague not reported beginning/end semester

315

Did the researchers Were establish confounders Were exact assumptions controlled in test statistic of analysis design values and p consistent addressed in levels Authors with data? analysis? presented? Kounenou et al. (2015) no no no Lim and Chan (2007) no no no Alayyar et al. (2012) no no yes Cengiz (2015) no no no Anderson and Maninger (2007) no no no Bai and Ertmer (2008) no yes no Willis (2015) no no no Lambert et al. (2008) consistent no no Jang (2008) no no yes Clark et al. (2015) adjusted no no Mizell (2016) no no no Karatas (2014) consistent no yes

316

APPENDIX S KNOWLEDGE RESEARCH DESIGN DATA

Was the need for Was the research Was the research Was the research the research well grounded in clearly clearly define the Authors stated? theory? described? population? C. Lee and Kim (2014) yes yes yes yes Alexander et al. (2014) yes yes yes yes Alayyar et al. (2012) yes yes yes yes Pheeraphan (2013) yes yes yes yes Lyublinskaya and Tournaki (2014) yes yes yes yes Blankson et al. (2010) yes no yes yes Cengiz (2015) yes yes yes yes Goodwin (2012) yes yes yes no Kohen and Kramarski (2012) yes yes yes yes Anderson and Maninger (2007) yes yes yes yes Mettas and Constantinou (2008) yes yes yes yes Lambert et al. (2008) yes yes yes yes Ward and Overall (2011) yes yes yes no Martinovic and Zhang (2012) yes yes yes yes Mizell (2016) yes yes yes yes Han et al. (2013) yes yes yes no Shah (2015) yes yes yes yes Kramarski and Michalsky (2009) yes yes yes yes Agyei and Voogt (2015) yes yes yes yes Mitchell (2013) yes yes yes no Schmidt et al. (2009) yes yes yes yes An et al. (2011) yes yes yes yes Shinas et al. (2015) yes yes yes yes Chai et al. (2010) yes yes yes yes Hur (2009) yes no yes no Duran and Fossum (2010) yes yes yes yes Abbitt (2011) yes yes yes yes Southall (2012) yes yes yes yes Starcic et al. (2016) yes yes yes no Mouza et al. (2014) yes yes yes yes Coffman (2013) yes yes yes yes Johnson (2012) yes yes yes yes Graham et al. (2012) yes yes yes yes Kafyulilo et al. (2015) yes yes yes no Sabo (2013) yes yes yes yes Agyei and Keengwe (2014) yes yes yes no Chai et al. (2011) yes yes yes yes Ersoy et al. (2016) yes yes yes no

317

Did the study Did the study address potential Did the study report any sub- bias in how the report on student group analyses course under participants that (i.e., gender, examination was declined Authors ethnicity, etc.)? selected? participation? C. Lee and Kim (2014) no no yes Alexander et al. (2014) yes no no Alayyar et al. (2012) yes no yes Pheeraphan (2013) no no yes Lyublinskaya and Tournaki (2014) yes yes yes Blankson et al. (2010) no yes no Cengiz (2015) no yes yes Goodwin (2012) no no yes Kohen and Kramarski (2012) no no yes Anderson and Maninger (2007) yes yes no Mettas and Constantinou (2008) no no yes Lambert et al. (2008) yes no no Ward and Overall (2011) no yes yes Martinovic and Zhang (2012) yes yes yes Mizell (2016) no yes yes Han et al. (2013) yes yes yes Shah (2015) no yes yes Kramarski and Michalsky (2009) no no no Agyei and Voogt (2015) no no no Mitchell (2013) no no no Schmidt et al. (2009) no no yes An et al. (2011) no no yes Shinas et al. (2015) no yes yes Chai et al. (2010) no no yes Hur (2009) no no no Duran and Fossum (2010) yes yes yes Abbitt (2011) no no yes Southall (2012) yes yes yes Starcic et al. (2016) yes yes no Mouza et al. (2014) no no no Coffman (2013) no no yes Johnson (2012) yes yes yes Graham et al. (2012) no no yes Kafyulilo et al. (2015) no no no Sabo (2013) yes no no Agyei and Keengwe (2014) no no no Chai et al. (2011) no no yes Ersoy et al. (2016) yes no no

318

Did the study authors report on the appropriateness of sample size for impact on statistical quality and/or What kind of sampling Authors conduct analyses for power? was used for the study? C. Lee and Kim (2014) not appropriate/reported non-probability Alexander et al. (2014) not appropriate/reported non-probability Alayyar et al. (2012) not appropriate/reported non-probability Pheeraphan (2013) not appropriate/reported non-probability Lyublinskaya and Tournaki (2014) defended non-probability Blankson et al. (2010) not appropriate/reported non-probability Cengiz (2015) not appropriate/reported non-probability Goodwin (2012) not appropriate/reported non-probability Kohen and Kramarski (2012) not appropriate/reported non-probability Anderson and Maninger (2007) not appropriate/reported non-probability Mettas and Constantinou (2008) not appropriate/reported non-probability Lambert et al. (2008) not appropriate/reported non-probability Ward and Overall (2011) not appropriate/reported non-probability Martinovic and Zhang (2012) not appropriate/reported non-probability Mizell (2016) not appropriate/reported non-probability Han et al. (2013) not appropriate/reported non-probability Shah (2015) not appropriate/reported non-probability Kramarski and Michalsky (2009) not appropriate/reported non-probability Agyei and Voogt (2015) not appropriate/reported non-probability Mitchell (2013) not appropriate/reported non-probability Schmidt et al. (2009) not appropriate/reported non-probability An et al. (2011) not appropriate/reported non-probability Shinas et al. (2015) power analysis non-probability Chai et al. (2010) not appropriate/reported non-probability Hur (2009) not appropriate/reported non-probability Duran and Fossum (2010) not appropriate/reported probability Abbitt (2011) not appropriate/reported non-probability Southall (2012) power analysis non-probability Starcic et al. (2016) not appropriate/reported non-probability Mouza et al. (2014) not appropriate/reported non-probability Coffman (2013) not appropriate/reported non-probability Johnson (2012) not appropriate/reported non-probability Graham et al. (2012) not appropriate/reported non-probability Kafyulilo et al. (2015) not appropriate/reported non-probability Sabo (2013) power analysis non-probability Agyei and Keengwe (2014) not appropriate/reported non-probability Chai et al. (2011) not appropriate/reported non-probability Ersoy et al. (2016) not appropriate/reported non-probability

319

Was there any management of pre- Was a comparison group used course differences between Authors and appropriate for the study? comparison groups? C. Lee and Kim (2014) no comparison group no Alexander et al. (2014) no comparison group no Alayyar et al. (2012) appropriate no Pheeraphan (2013) no comparison group no Lyublinskaya and Tournaki (2014) no comparison group yes Blankson et al. (2010) no comparison group yes Cengiz (2015) no comparison group no Goodwin (2012) appropriate no Kohen and Kramarski (2012) no comparison group no Anderson and Maninger (2007) no comparison group no Mettas and Constantinou (2008) no comparison group no Lambert et al. (2008) no comparison group yes Ward and Overall (2011) appropriate no Martinovic and Zhang (2012) no comparison group yes Mizell (2016) no comparison group no Han et al. (2013) appropriate yes Shah (2015) no comparison group no Kramarski and Michalsky (2009) appropriate yes Agyei and Voogt (2015) appropriate no Mitchell (2013) no comparison group no Schmidt et al. (2009) no comparison group no An et al. (2011) no comparison group no Shinas et al. (2015) no comparison group no Chai et al. (2010) no comparison group no Hur (2009) no comparison group no Duran and Fossum (2010) no comparison group yes Abbitt (2011) no comparison group no Southall (2012) no comparison group yes Starcic et al. (2016) no comparison group no Mouza et al. (2014) no comparison group no Coffman (2013) no comparison group no Johnson (2012) appropriate yes Graham et al. (2012) no comparison group no Kafyulilo et al. (2015) no comparison group no Sabo (2013) appropriate yes Agyei and Keengwe (2014) no comparison group no Chai et al. (2011) no comparison group no Ersoy et al. (2016) no comparison group no

320

Did the researchers randomize participants, instructor, assessor measuring outcome, and data Would the course be Authors analyst? replicable by others? C. Lee and Kim (2014) no/unknown highly Alexander et al. (2014) no/unknown some Alayyar et al. (2012) no/unknown some Pheeraphan (2013) no/unknown some Lyublinskaya and Tournaki (2014) no/unknown highly Blankson et al. (2010) no/unknown some Cengiz (2015) no/unknown some Goodwin (2012) no/unknown some Kohen and Kramarski (2012) no/unknown some Anderson and Maninger (2007) no/unknown some Mettas and Constantinou (2008) no/unknown highly Lambert et al. (2008) no/unknown highly Ward and Overall (2011) no/unknown some Martinovic and Zhang (2012) no/unknown some Mizell (2016) no/unknown some Han et al. (2013) no/unknown some Shah (2015) no/unknown some Kramarski and Michalsky (2009) single some Agyei and Voogt (2015) no/unknown some Mitchell (2013) no/unknown some Schmidt et al. (2009) no/unknown some An et al. (2011) no/unknown some Shinas et al. (2015) no/unknown some Chai et al. (2010) no/unknown some Hur (2009) no/unknown some Duran and Fossum (2010) multiple some Abbitt (2011) no/unknown some Southall (2012) no/unknown some Starcic et al. (2016) no/unknown highly Mouza et al. (2014) no/unknown highly Coffman (2013) no/unknown some Johnson (2012) single highly Graham et al. (2012) no/unknown some Kafyulilo et al. (2015) no/unknown some Sabo (2013) single some Agyei and Keengwe (2014) no/unknown some Chai et al. (2011) no/unknown some Ersoy et al. (2016) no/unknown some

321

Did the study To what extent define was the course Was the integrity technologies design consistent of the course related to the Authors with theory? maintained? course? C. Lee and Kim (2014) high no/unknown vague Alexander et al. (2014) not/unknown yes vague Alayyar et al. (2012) not/unknown maybe no Pheeraphan (2013) high yes vague Lyublinskaya and Tournaki (2014) high yes detailed Blankson et al. (2010) low yes detailed Cengiz (2015) not/unknown no/unknown vague Goodwin (2012) low maybe vague Kohen and Kramarski (2012) high yes vague Anderson and Maninger (2007) not/unknown no/unknown vague Mettas and Constantinou (2008) high yes vague Lambert et al. (2008) high yes detailed Ward and Overall (2011) not/unknown yes no Martinovic and Zhang (2012) not/unknown no/unknown no Mizell (2016) not/unknown no/unknown no Han et al. (2013) not/unknown yes no Shah (2015) high yes detailed Kramarski and Michalsky (2009) high yes vague Agyei and Voogt (2015) high yes vague Mitchell (2013) not/unknown maybe no Schmidt et al. (2009) not/unknown no/unknown no An et al. (2011) not/unknown no/unknown no Shinas et al. (2015) high yes no Chai et al. (2010) high yes no Hur (2009) low maybe vague Duran and Fossum (2010) high yes vague Abbitt (2011) not/unknown no/unknown no Southall (2012) not/unknown no/unknown no Starcic et al. (2016) low yes vague Mouza et al. (2014) high yes vague Coffman (2013) not/unknown no/unknown no Johnson (2012) high yes detailed Graham et al. (2012) low yes vague Kafyulilo et al. (2015) high yes no Sabo (2013) low yes vague Agyei and Keengwe (2014) not/unknown no/unknown no Chai et al. (2011) high yes vague Ersoy et al. (2016) low yes vague

322

Was study attrition prevented and/or What was the measurement period Authors reported? between measures? C. Lee and Kim (2014) not reported multiple weeks Alexander et al. (2014) not reported one class Alayyar et al. (2012) not reported beginning/end semester Pheeraphan (2013) not reported beginning/end semester Lyublinskaya and Tournaki (2014) not reported multiple weeks Blankson et al. (2010) not reported beginning/end semester Cengiz (2015) not reported beginning/end semester Goodwin (2012) not prevented beginning/end semester Kohen and Kramarski (2012) not reported beginning/end semester Anderson and Maninger (2007) not reported beginning/end semester Mettas and Constantinou (2008) not reported beginning/end semester Lambert et al. (2008) not reported beginning/end semester Ward and Overall (2011) not reported beginning/end semester Martinovic and Zhang (2012) not prevented beginning/end semester Mizell (2016) not reported beginning/end semester Han et al. (2013) not reported multiple weeks Shah (2015) not prevented beginning/end semester Kramarski and Michalsky (2009) not reported beginning/end semester Agyei and Voogt (2015) not reported multiple weeks Mitchell (2013) not reported beginning/end semester Schmidt et al. (2009) not prevented beginning/end semester An et al. (2011) not prevented beginning/end semester Shinas et al. (2015) not prevented beginning/end semester Chai et al. (2010) not prevented beginning/end semester Hur (2009) not reported beginning/end semester Duran and Fossum (2010) not reported beginning/end semester Abbitt (2011) not reported beginning/end semester Southall (2012) not prevented beginning/end semester Starcic et al. (2016) not reported beginning/end semester Mouza et al. (2014) not prevented beginning/end semester Coffman (2013) not reported beginning/end semester Johnson (2012) not reported beginning/end semester Graham et al. (2012) not reported beginning/end semester Kafyulilo et al. (2015) not reported beginning/end semester Sabo (2013) not reported beginning/end semester Agyei and Keengwe (2014) not reported unknown Chai et al. (2011) not prevented beginning/end semester Ersoy et al. (2016) not reported beginning/end semester

323

Did the researchers establish Were confounders Were exact test statistic assumptions of analysis controlled in design values and p levels Authors consistent with data? addressed in analysis? presented? C. Lee and Kim (2014) no no yes Alexander et al. (2014) no no no Alayyar et al. (2012) no no yes Pheeraphan (2013) adjusted no no Lyublinskaya and Tournaki (2014) no no yes Blankson et al. (2010) adjusted no no Cengiz (2015) no no no Goodwin (2012) no no no Kohen and Kramarski (2012) adjusted no no Anderson and Maninger (2007) no no no Mettas and Constantinou (2008) no no yes Lambert et al. (2008) consistent no no Ward and Overall (2011) no no yes Martinovic and Zhang (2012) consistent no no Mizell (2016) no no no Han et al. (2013) no no no Shah (2015) no no yes Kramarski and Michalsky (2009) no no no Agyei and Voogt (2015) no no no Mitchell (2013) no no yes Schmidt et al. (2009) no no no An et al. (2011) consistent no no Shinas et al. (2015) no no no Chai et al. (2010) no no no Hur (2009) no no yes Duran and Fossum (2010) no no yes Abbitt (2011) no no no Southall (2012) adjusted yes yes Starcic et al. (2016) no no no Mouza et al. (2014) no no yes Coffman (2013) no adjustment no no Johnson (2012) consistent yes no Graham et al. (2012) no no no Kafyulilo et al. (2015) adjusted no yes Sabo (2013) no yes no Agyei and Keengwe (2014) no no no Chai et al. (2011) no no no Ersoy et al. (2016) no no yes

324

LIST OF REFERENCES

AECT. (2008). Definition. In A. Januszewski & M. Molenda (Eds.), Educational technology: A definition with commentary: Routledge.

Aesaert, K., Van Nijlen, D., Vanderlinde, R., & van Braak, J. (2014). Direct measures of digital information processing and communication skills in primary education: Using item response theory for the development and validation of an ICT competence scale. Computers & Education, 76, 168-181.

Agyei, D. D., & Keengwe, J. (2014). Using technology pedagogical content knowledge development to enhance learning outcomes. Education and Information technologies, 19(1), 155-171.

Aiken, L. (2002). Attitudes and related psychosocial constructs: Theories, assessment, and research: Sage.

Alayyar, G. M., Fisser, P., & Voogt, J. (2012). Developing technological pedagogical content knowledge in pre-service science teachers: Support from blended learning. Australasian Journal of Educational Technology, 28(8).

Alexander, C., Knezek, G., Christensen, R., Tyler-Wood, T., & Bull, G. (2014). The impact of project-based learning on pre-service teachers’ technology attitudes and skills. Journal of Computers in Mathematics and Science Teaching, 33(3), 257-282.

Allport, G. W. (1967). Attitudes. In M. Fishbein (Ed.), Readings in attitude theory and measurement (pp. 1-14). New York, NY: John Wiley & Sons, Inc.

An, Y. J., & Reigeluth, C. (2011). Creating technology-enhanced, learner-centered classrooms: K–12 teachers’ beliefs, perceptions, barriers, and support needs. Journal of Digital Learning in Teacher Education, 28(2), 54-62.

Anderson, L. W., Krathwohl, D. R., & Bloom, B. S. (2001). A taxonomy for learning, teaching, and assessing: A revision of Bloom's taxonomy of educational objectives: Allyn & Bacon.

Anderson, S. E., & Maninger, R. M. (2007). Preservice teachers' abilities, beliefs, and intentions regarding technology integration. Journal of Educational Computing Research, 37(2), 151-172.

Angeli, C. (2005). Transforming a teacher education method course through technology: Effects on preservice teachers’ technology competency. Computers & Education, 45(4), 383- 398.

325

Angers, J., & Machtmes, K. L. (2005). An ethnographic-case study of beliefs, context factors, and practices of teachers integrating technology. The Qualitative Report, 10(4), 771-794.

Apple. (2008). Apple classrooms of tomorrow—today learning in the 21st Century: Background information. Retrieved from http://education.apple.com/acot2/global/files/ACOT2_Background.pdf

Archambault, L., & Crippen, K. (2006). The preparation and perspective of online K-12 teachers in Nevada. Paper presented at the Proceedings of the World Conference on E-Learning in Corporate, Government, Healthcare, and Higher Education.

Attitude. (Ed.) (2015) APA Dictionary of Psychology (2nd ed.). Washington, D.C.: American Psychological Association.

Attitude object. (Ed.) (2015) APA Dictionary of Psychology (2nd ed.). Washington, D.C.: American Psychological Association.

Avramidis, E., & Norwich, B. (2002). Teachers' attitudes towards integration/inclusion: a review of the literature. European Journal of Special Needs Education, 17(2), 129-147.

Bakir, N. (2016). Technology and Teacher Education: A Brief Glimpse of the Research and Practice that Have Shaped the Field. TechTrends, 60(1), 21-29.

Bandura, A. (1986). Social foundations of thought and action: A social cognitive theory. Englewood Cliffs, NJ: Prentice-Hall.

Bandura, A. (2006). Guide for constructing self-efficacy scales. Self-efficacy beliefs of adolescents, 5(307-337).

Barron, A. E., Kemker, K., Harmes, C., & Kalaydjian, K. (2003). Large-scale research study on technology in K–12 schools: Technology integration as it relates to the National Technology Standards. Journal of Research on Technology in Education, 35(4), 489-507.

Becker, B. J. (2005). Failsafe N or file-drawer number. Publication bias in meta-analysis: Prevention, assessment and adjustments, 111-125.

Belief. (Ed.) (2015) APA Dictionary of Psychology (2nd ed.). Washington, D.C.: American Psychological Association.

Bingimlas, K. A. (2009). Barriers to the successful integration of ICT in teaching and learning environments: A review of the literature. Eurasia Journal of Mathematics, Science & Technology Education, 5(3), 235-245.

326

Binkley, M., Erstad, O., Herman, J., Raizen, S., Ripley, M., Miller-Ricci, M., & Rumble, M. (2012). Defining twenty-first century skills Assessment and teaching of 21st century skills (pp. 17-66): Springer.

Birch, A., & Irvine, V. (2009). Preservice teachers’ acceptance of ICT integration in the classroom: applying the UTAUT model. Educational Media International, 46(4), 295- 315.

Blackwell, C. K., Lauricella, A. R., Wartella, E., Robb, M., & Schomburg, R. (2013). Adoption and use of technology in early education: The interplay of extrinsic barriers and teacher attitudes. Computers & Education, 69, 310-319.

Bodur, H. O., Brinberg, D., & Coupey, E. (2000). Belief, affect, and attitude: Alternative models of the determinants of attitude. Journal of Consumer Psychology, 9(1), 17-28.

Borenstein, M., Hedges, L. V., Higgins, J. P. T., & Rothstein, H. R. (2009). Introduction to meta- analysis. Chichester, England: Wiley.

Brush, T., Igoe, A., Brinkerhoff, J., Glazewski, K., Heng-Yu, K., & Colette Smith, T. (2001). Lessons from the field: Integrating technology into preservice teacher education. Journal of Computing in Teacher Education, 17(4), 16-20.

Buckenmeyer, J. A. (2010). Beyond computers in the classroom: Factors related to technology adoption to enhance teaching and learning. Contemporary Issues in Education Research, 3(4), 27.

Campion, M. (1989). Technophilia and technophobia. Australian Journal of Educational Technology, 5(1), 23-36.

Cengiz, C. (2015). The development of TPACK, Technology Integrated Self-Efficacy and Instructional Technology Outcome Expectations of pre-service physical education teachers. Asia-Pacific Journal of Teacher Education, 43(5), 411-422.

Christensen, R., & Knezek, G. (1998). Parallel forms for measuring teacher's attitudes toward computers. In S. McNeil, J. Price, S. Boger-Mehall, B. Robin, & J. Willis (Eds.), Technology and teacher education annual 1998 (Vol. 2, pp. 820-824). Charlottesville, VA: Association for the Advancement of Computing in Education.

Christensen, R., & Knezek, G. (2001). Instruments for assessing the impact of technology in education. Computers in the Schools, 18(2-3), 5-25.

327

Chua, S. L., Chen, D.-T., & Wong, A. F. (1999). Computer anxiety and its correlates: a meta- analysis. Computers in Human Behavior, 15(5), 609-623.

Chuang, H.-H., Thompson, A., & Schmidt, D. (2003). Faculty technology mentoring programs: Major trends in the literature. Journal of Computing in Teacher Education, 19(4), 101- 106.

Clark, C., Zhang, S., & Strudler, N. (2015). Teacher candidate technology integration: For student learning or instruction? Journal of Digital Learning in Teacher Education, 31(3), 93-106.

Clark, R. E. (1983). Reconsidering research on learning from media. Review of educational research, 53(4), 445-459.

Clore, G. L., & Schnall, S. (2005). The influence of affect on attitude. The handbook of attitudes, 437-489.

Cohen, J. (1960). A coefficient of agreement for nominal scales. Educational and Psychological measurement, 20(1), 37-46.

Committee on Information Technology Literacy. (1999). Being fluent with information technology: National Academy Press.

Conn, V. S., & Rantz, M. J. (2003). Research methods: Managing primary study quality in meta‐ analyses. Research in Nursing & Health, 26(4), 322-333.

Cooper, H. (2017). Research synthesis and meta-analysis: A step-by-step approach (5th ed.). Thousand Oaks, CA: Sage.

Cooper, H., Hedges, L. V., & Valentine, J. C. (Eds.). (2009). The handbook of research synthesis and meta-analysis (2nd ed.). New York, NY: Russell Sage Foundation.

Crocker, L., & Algina, J. (1986). Introduction to classical and modern test theory: Holt, Rinehart, and Winston, Inc.

Dale, E. (1946). Audio-visual methods in teaching Audio-visual methods in teaching (pp. 2-66). New York: Dryden Press.

Davis, F. D. (1993). User acceptance of information technology: System characteristics, user perceptions and behavioral impacts. International Journal of Man-Machine Studies, 38(3).

Davis, F. D., Bagozzi, R. P., & Warshaw, P. R. (1989). User acceptance of computer technology: a comparison of two theoretical models. Management science, 35(8), 982-1003.

Dawley, L., & Dede, C. (2014). Situated learning in virtual worlds and immersive simulations Handbook of research on educational communications and technology (pp. 723-734): Springer.

328

Del Re, A. C., & Hoyt W. T. (2014). MAd: Meta-Analysis with Mean Differences. R package version 0.8-2 [Software]. Available from http://cran.r-project.org/web/packages/MAd

Dias, L. B. (1999). Integrating technology. Learning and Leading with technology, 27, 10-13.

Dickersin, K., Rothstein, H., Sutton, A., & Borenstein, M. (2005). Publication bias: Recognizing the problem, understanding its origins and scope, and preventing harm. Publication bias in meta-analysis: Prevention, assessment and adjustments, 11-33.

Dunst, C., Hamby, D., & Trivette, C. (2004). Guidelines for calculating effect sizes for practice- based research syntheses. Centerscope, 3(1), 1-10.

Duval, S. (2005). The trim and fill method. Publication bias in meta-analysis: Prevention, assessment, and adjustments, 127-144.

Duval, S., & Tweedie, R. (2000a). A nonparametric “trim and fill” method of accounting for publication bias in meta-analysis. Journal of the American Statistical Association, 95(449), 89-98.

Duval, S., & Tweedie, R. (2000b). Trim and fill: a simple funnel‐plot–based method of testing and adjusting for publication bias in meta‐analysis. Biometrics, 56(2), 455-463.

Eagly, A. H., & Chaiken, S. (1993). The psychology of attitudes: Harcourt Brace Jovanovich College Publishers.

Ennis, L. A. (2005). The evolution of technostress. Computers in libraries, 25(8), 10-12.

Enochsson, A., & Rizza, C. (2009). ICT in initial teacher training: Research review. OECD Education Working Papers, No. 38. Retrieved from http://dx.doi.org/10.1787/220502872611

Ertmer, P. A. (1999). Addressing first-and second-order barriers to change: Strategies for technology integration. Educational Technology Research and Development, 47(4), 47- 61.

Ertmer, P. A. (2005). Teacher pedagogical beliefs: The final frontier in our quest for technology integration? Educational Technology Research and Development, 53(4), 25-39.

Ertmer, P. A., & Newby, T. J. (1993). Behaviorism, cognitivism, constructivism: Comparing critical features from an instructional design perspective. Performance improvement quarterly, 6 (4), 50‐72.

Ertmer, P. A., & Ottenbreit-Leftwich, A. (2013). Removing obstacles to the pedagogical changes required by Jonassen's vision of authentic technology-enabled learning. Computers & Education, 64, 175-182.

329

Ertmer, P. A., Ottenbreit-Leftwich, A., & York, C. S. (2006). Exemplary technology-using teachers: Perceptions of factors influencing success. Journal of Computing in Teacher Education, 23(2), 55-61.

Ertmer, P. A., & Ottenbreit-Leftwich, A. T. (2010). Teacher technology change: How knowledge, confidence, beliefs, and culture intersect. Journal of Research on Technology in Education, 42(3).

Ertmer, P. A., Ottenbreit-Leftwich, A. T., Sadik, O., Sendurur, E., & Sendurur, P. (2012). Teacher beliefs and technology integration practices: A critical relationship. Computers & Education, 59(2), 423-435.

Fishbein, M., & Ajzen, I. (1975). Belief, attitude, intention and behavior: an introduction to theory and research. Reading, MA: Addison-Wesley.

Forgas, J. P. (2008). The role of affect in attitudes and attitude change. In W. D. Crano & R. Prislin (Eds.), Attitudes and attitude change (pp. 131-158). New York, NY: Psychology Press.

Franklin, T., Turner, S., Kariuki, M., & Duran, M. (2001). Mentoring overcomes barriers to technology integration. Journal of Computing in Teacher Education, 18(1), 26-31.

Gagné, R. M. (1970). The conditions of learning.

Gersten, R., & Edyburn, D. (2007). Defining quality indicators for special education technology research. Journal of Special Education Technology, 22(3), 3-18.

Ginsburg, M. (2009). EQUIP2 State-of-the-Art Knowledge in Education: Washington, DC: United States Agency for International Development.

Goktas, Y., Yildirim, S., & Yildirim, Z. (2009). Main barriers and possible enablers of ICTs integration into pre-service teacher education programs. Educational Technology & Society, 12(1), 193-204.

Gronseth, S., Brush, T., Ottenbreit-Leftwich, A., Strycker, J., Abaci, S., Easterling, W., . . . Leusen, P. v. (2010). Equipping the next generation of teachers: Technology preparation and practice. Journal of Digital Learning in Teacher Education, 27(1), 30-36.

Harmes, J. C., Barron, A., & Kemker, K. (2007). Development and implementation of an online tool to assess teacher technology skills. Technology and teacher education annual, 18(2), 802.

Harmes, J. C., Welsh, J. L., & Winkelman, R. J. (2016). A framework for defining and evaluating technology integration in the Instruction of real-world skills. Handbook of Research on Technology Tools for Real-World Skill Development, 137-162.

330

Harris, J., & Hofer, M. (2009). Grounded tech integration: An effective approach based on content, pedagogy, and teacher planning. Learning & Leading with Technology, 37(2), 22-25.

Hasan, B. (2003). The influence of specific computer experiences on computer self-efficacy beliefs. Computers in Human Behavior, 19(4), 443-450.

Hedges, L. V. (1981). Distribution theory for Glass's estimator of effect size and related estimators. Journal of Educational and Behavioral Statistics, 6(2), 107-128.

Hedges, L. V., & Pigott, T. D. (2001). The power of statistical tests in meta-analysis. Psychological methods, 6(3), 203.

Heinssen, R. K., Glass, C. R., & Knight, L. A. (1987). Assessing computer anxiety: Development and validation of the computer anxiety rating scale. Computers in Human Behavior, 3(1), 49-59.

Heo, M. (2009). Digital storytelling: An empirical study of the impact of digital storytelling on pre-service teachers' self-efficacy and dispositions towards educational technology. Journal of Educational Multimedia and Hypermedia, 18(4), 405.

Hermans, R., Tondeur, J., van Braak, J., & Valcke, M. (2008). The impact of primary school teachers’ educational beliefs on the classroom use of computers. Computers & Education, 51(4), 1499-1509.

Hew, K. F., & Brush, T. (2007). Integrating technology into K-12 teaching and learning: Current knowledge gaps and recommendations for future research. Educational Technology Research and Development, 55(3), 223-252.

Higgins, J., & Thompson, S. G. (2002). Quantifying heterogeneity in a meta‐analysis. Statistics in medicine, 21(11), 1539-1558.

Hofer, M., & Grandgenett, N. (2012). TPACK development in teacher education: A longitudinal study of preservice teachers in a secondary MA Ed. program. Journal of Research on Technology in Education, 45(1), 83-106.

Hohlfeld, T. N., Ritzhaupt, A. D., & Barron, A. E. (2010). Development and Validation of the Student Tool for Technology Literacy (ST2L). Journal of Research on Technology in Education, 42(4), 361-389.

Hohlfeld, T. N., Ritzhaupt, A. D., Barron, A. E., & Kemker, K. (2008). Examining the digital divide in K-12 public schools: Four-year trends for supporting ICT literacy in Florida. Computers & Education, 51(4), 1648-1663.

Hoskin, R. (2012). The dangers of self-report. Science Brainwaves.

Hudiburg, R. A. (1995). Psychology of computer use: XXXIV. The computer hassles scale: Subscales, norms, and reliability. Psychological Reports, 77(3), 779-782.

331

Huedo-Medina, T. B., Sánchez-Meca, J., Marín-Martínez, F., & Botella, J. (2006). Assessing heterogeneity in meta-analysis: Q statistic or I² index? Psychological methods, 11(2), 193.

Husbands, C., & Pearce, J. (2012). What makes great pedagogy? Nine claims from research.

Inan, F. A., & Lowther, D. L. (2010). Factors affecting technology integration in K-12 classrooms: A path model. Educational Technology Research and Development, 58(2), 137-154.

International Society for Technology in Education. (2017a). The ISTE story. Retrieved from http://www.iste.org/about/iste-story

International Society for Technology in Education. (2017b). Standards. Retrieved from http://www.iste.org/standards/standards

International Society for Technology in Education. (2017c). ISTE standards for students 2007. Retrieved from http://www.iste.org/standards/standards/standards-for-students

International Society for Technology in Education. (2017d). ISTE standards for teachers 2007. Retrieved from http://www.iste.org/standards/iste-standards/standards-for-teachers

International Society for Technology in Education. (2017e). Essential conditions. Retrieved from http://www.iste.org/standards/essential-conditions

International Society for Technology in Education. (2017f). ISTE standards faqs. Retrieved from http://www.iste.org/standards/standards/iste-standards-2016-faq

International Society for Technology in Education. (2017g). ISTE standards and the common core. Retrieved from https://www.iste.org/standards/standards-in-action/common-core

International Society for Technology in Education. (2017h). ISTE standards for students 2016. Retrieved from https://www.iste.org/standards/standards/for-students-2016

Ioannidis, J., & Lau, J. (1999). Pooling research results: benefits and limitations of meta- analysis. The Joint Commission journal on quality improvement, 25(9), 462-469.

Johnson, L., Adams-Becker, S., Estrada, V., & Freeman, A. (2015). NMC Horizon Report: 2015 K-12 Edition. Austin, Texas: The New Media Consortium.

Karatas, I. (2014). Changing pre-service mathematics teachers’ beliefs about using computers for teaching and learning mathematics: the effect of three different models. European Journal of Teacher Education, 37(3), 390-405.

Katz, I. R. (2007). Testing information literacy in digital environments: ETS's iSkills assessment. Information technology and Libraries, 26(3), 3.

332

Kay, R. H. (2006). Evaluating strategies used to incorporate technology into preservice education: A review of the literature. Journal of Research on Technology in Education, 38(4), 383-408.

Keengwe, J., Onchwari, G., & Wachira, P. (2008). Computer technology integration and student learning: Barriers and promise. Journal of Science Education and Technology, 17(6), 560-565.

Kelly, M. G. (Ed.) (2002). National educational technology standards for teachers: Preparing teachers to use technology. Eugene, OR: ISTE.

Kinzie, M. B., & Delcourt, M. A. (1991). Computer technologies in teacher education: The measurement of attitudes and self-efficacy.

Kirshstein, R., Birman, B., Quinones, S., Levin, D., Stephens, M., & Loy, N. (2000). The First- Year Implementation of the Technology Literacy Challenge Fund in Five States.

Knezek, G., & Christensen, R. (1998). Internal consistency reliability for the teachers’ attitudes toward information technology (TAT) questionnaire. Paper presented at the Proceedings of the Society for Information Technology in Teacher Education Annual Conference.

Koehler, M., & Mishra, P. (2005). Teachers learning technology by design. Journal of Computing in Teacher Education, 21(3), 94-102.

Koehler, M., & Mishra, P. (2009). What is technological pedagogical content knowledge (TPACK)? Contemporary Issues in Technology and Teacher Education, 9(1), 60-70.

Koehler, M., Mishra, P., & Yahya, K. (2007). Tracing the development of teacher knowledge in a design seminar: Integrating content, pedagogy and technology. Computers & Education, 49(3), 740-762.

Koehler, M., Shin, T. S., & Mishra, P. (2011). How do we measure TPACK? Let me count the ways. Educational technology, teacher knowledge, and classroom impact: A research handbook on frameworks and approaches, 16-31.

Koh, J. H., & Divaharan, S. (2011). Developing pre-service teachers' technology integration expertise through the TPACK-developing instructional model. Journal of Educational Computing Research, 44(1), 35-58.

Kolb, A. Y., & Kolb, D. A. (2005). Learning styles and learning spaces: Enhancing experiential learning in higher education. Academy of management learning & education, 4(2), 193- 212.

333

Kozma, R. B. (1991). Learning with media. Review of educational research, 61(2), 179-211.

Krathwohl, D., Bloom, B., & Masia, B. (1964). A taxonomy of educational objectives: Handbook II. The affective domain.

Lankshear, C., & Knobel, M. (2008). Digital literacies: Concepts, policies and practices (Vol. 30): Peter Lang.

Lee, C., & Kim, C. (2014). An implementation study of a TPACK-based instructional design model in a technology integration course. Educational Technology Research and Development, 62(4), 437-460.

Lee, M. H., & Tsai, C. C. (2010). Exploring teachers’ perceived self efficacy and technological pedagogical content knowledge with respect to educational use of the World Wide Web. Instructional science, 38(1), 1-21.

Lee, Y., & Lee, J. (2014). Enhancing pre-service teachers' self-efficacy beliefs for technology integration through lesson planning practice. Computers & Education, 73, 121-128.

Lei, J. (2009). Digital natives as preservice teachers: What technology preparation is needed? Journal of Computing in Teacher Education, 25(3), 87-97.

Liao, Y., & Bright, G. (1993). Meta-analysis in technology. Approaches to research on teacher education and technology, 93-100.

Lim, C. P., & Chan, B. C. (2007). MicroLESSONS in teacher education: Examining pre-service teachers’ pedagogical beliefs. Computers & Education, 48(3), 474-494.

Lim, C. P., & Khine, M. S. (2006). Managing teachers' barriers to ICT integration in Singapore schools. Journal of Technology and Teacher Education, 14(1), 97.

Lin, C. A. (2003). An interactive communication technology adoption model. Communication Theory, 13(4), 345-365.

Lipsey, M. W., & Wilson, D. (2000). Practical meta-analysis (applied social research methods).

Liu, F., Ritzhaupt, A. D., Dawson, K., & Barron, A. E. (2016). Explaining technology integration in K-12 classrooms: A multilevel path analysis model. Educational Technology Research and Development, 1-19. doi:10.1007/s11423-016-9487-9

Lowther, D. L., Inan, F. A., Strahl, J. D., & Ross, S. M. (2008). Does technology integration “work” when key barriers are removed? Educational Media International, 45(3), 195- 213.

334

Loyd, B. H., & Loyd, D. E. (1985). The reliability and validity of an instrument for the assessment of computer attitudes. Educational and Psychological measurement, 45(4), 903-908.

Ma, J., & Nickerson, J. V. (2006). Hands-on, simulated, and remote laboratories: A comparative literature review. ACM Computing Surveys (CSUR), 38(3), 7.

Maio, G., & Haddock, G. (2014). The psychology of attitudes and attitude change (2nd ed.): Sage.

Marsh, K. L., & Wallace, H. M. (2005). The influence of attitudes on beliefs: Formation and change. The handbook of attitudes, 369-395.

Marvin, E. D., Lowther, D. L., & Ross, S. M. (2002). Technology skills assessment. Memphis, TN: Center for Research in Educational Policy, The University of Memphis.

Mayer, R. E. (2005). Cognitive theory of multimedia learning. The Cambridge Handbook of Multimedia Learning.

McHugh, M. L. (2012). Interrater reliability: the kappa statistic. Biochemia medica, 22(3), 276- 282.

Merrill, D. M. (2002). First principles of instruction. Educational Technology Research and Development, 50(3), 43-59. doi:10.1007/BF02505024

Meuter, M. L., Ostrom, A. L., Bitner, M. J., & Roundtree, R. (2003). The influence of technology anxiety on consumer use and experiences with self-service technologies. Journal of Business Research, 56(11), 899-906.

Milman, N. B., & Molebash, P. E. (2008). A longitudinal assessment of teacher education students' confidence toward using technology. Journal of Educational Computing Research, 38(2), 183-200.

Mishra, P., & Koehler, M. (2006). Technological pedagogical content knowledge: A framework for teacher knowledge. The Teachers College Record, 108(6), 1017-1054.

Mizell, S. (2016). Factors affecting early adoption of technology.

Moher, D., Liberati, A., Tetzlaff, J., Altman, D. G., & Group, P. (2009). Preferred reporting items for systematic reviews and meta-analyses: the PRISMA statement. PLoS med, 6(7), e1000097.

Morris, S. B., & DeShon, R. P. (2002). Combining effect size estimates in meta-analysis with repeated measures and independent-groups designs. Psychological methods, 7(1), 105.

335

Morrison, G. R., Ross, S. M., Kalman, H. K., & Kemp, J. E. (2013). Designing effective instruction (7th ed. ed.). Hoboken, NJ: John Wiley & Sons, Inc.

National Assessment Governing Board. (2014). Technology and engineering literacy framework for the 2014 national assessment of educational progress. Washington, D.C.: National Assessment Governing Board Retrieved from https://www.nagb.org/content/nagb/assets/documents/publications/frameworks/technolog y/2014-technology-framework.pdf.

National Governors Association Center for Best Practices. (2010). Common core state standards. Retrieved from http://www.corestandards.org/

NGA. (2017). FAQ. Retrieved from https://www.nga.org/cms/home/about/faq.html

Niess, M. L. (2008). Guiding preservice teachers in developing TPCK. In M. C. Herring, M. Koehler, & P. Mishra (Eds.), Handbook of technological pedagogical content knowledge (TPCK) for educators (1 ed., pp. 223-250). New York, NY: Taylor & Francis.

Nilson, L. B. (2016). Teaching at its best: A research-based resource for college instructors (3rd ed.). San Francisco, CA: John Wiley & Sons, Inc.

OECD. (2009). Creating effective teaching and learning environments: First results from TALIS. Paris, France: OECD.

Orwin, R. G. (1983). A fail-safe N for effect size in meta-analysis. Journal of educational statistics, 8(2), 157-159.

Osgood, C. E., Suci, G. J., & Tannenbaum, P. H. (1957). The measurement of meaning. Urbana, IL: University of Illinois Press.

Ottenbreit-Leftwich, A. T., Glazewski, K. D., Newby, T. J., & Ertmer, P. A. (2010). Teacher value beliefs associated with using technology: Addressing professional and student needs. Computers & Education, 55(3), 1321-1335.

P21. (2009). P21 framework for 2st century learning. Retrieved from http://www.p21.org/our- work/p21-framework

Pajares, M. F. (1992). Teachers’ beliefs and educational research: Cleaning up a messy construct. Review of educational research, 62(3), 307-332.

Pamuk, S., Ergun, M., Cakir, R., Yilmaz, H. B., & Ayas, C. (2013). Exploring relationships among TPACK components and development of the TPACK instrument. Education and Information technologies, 20(2), 241-263.

336

Parente, S. L., & Prescott, E. C. (1994). Barriers to technology adoption and development. Journal of political Economy, 102(2), 298-321.

Park, H. S., Gilbreath, J., Lawson, D., & Williams, H. E. (2010). Exploring the Factors to Determine the Competence of Technology Integration for Teacher Candidates. Adaptation, Resistance and Access to Instructional Technologies: Assessing Future Trends In Education: Assessing Future Trends In Education, 355.

Pigott, T. (2012). Advances in meta-analysis: Springer Science & Business Media.

Pope, M., Hare, D., & Howardy, E. (2002). Technology integration: Closing the gap between what preservice teachers are taught to do and what they can do. Journal of Technology and Teacher Education, 10(2), 191-204.

Quinn, C. N., & Connor, M. L. (2005). Engaging learning : designing e-learning simulation games. San Francisco, CA: Jossey-Bass.

Ragu-Nathan, T., Tarafdar, M., Ragu-Nathan, B. S., & Tu, Q. (2008). The consequences of technostress for end users in organizations: Conceptual development and empirical validation. Information Systems Research, 19(4), 417-433.

Reeves, T. C. (1998). The impact of media and technology in schools. Journal of The Journal of Art and Design Education, 2, 58-63.

Reid, P. (2014). Categories for barriers to adoption of instructional technologies. Education and Information technologies, 19(2), 383-407.

Reiser, R. A. (2001). A history of instructional design and technology: Part I: A history of instructional media. Educational Technology Research and Development, 49(1), 53-64. doi:10.1007/BF02504506

Richardson, V. (1996). The role of attitudes and beliefs in learning to teach. In J. Sikula (Ed.), Handbook of research on teacher education (2nd ed., pp. 102-119). New York, NY: Simon & Schuster Macmillan.

Ringstaff, C., Yocam, K., & Marsh, J. (1996). Integrating technology into classroom instruction: An assessment of the impact of the ACOT Teacher Development Center Project (ACOT Report #22). Retrieved from http://www.apple.com/euro/pdfs/acotlibrary/rpt22.pdf

Ritzhaupt, A. D., Dawson, K., & Cavanaugh, C. (2012). An investigation of factors influencing student use of technology in K-12 classrooms using path analysis. Journal of Educational Computing Research, 46(3), 229-254.

Ritzhaupt, A. D., Huggins-Manley, A. C., Ruggles, K., & Wilson, M. (2016). Validation of the survey of pre-service teachers’ knowledge of teaching and technology: A multi- institutional sample. Journal of Digital Learning in Teacher Education, 32(1), 26-37.

337

Rodgers, C. (2002). Defining reflection: Another look at John Dewey and reflective thinking. Teachers college record, 104(4), 842-866.

Ropp, M. M. (1999). Exploring individual characteristics associated with learning to use computers in preservice teacher preparation. Journal of Research on Computing in Education, 31(4), 402-424.

Rosen, L. D., & Weil, M. M. (1995a). Computer anxiety: A cross-cultural comparison of university students in ten countries. Computers in Human Behavior, 11(1), 45-64.

Rosen, L. D., & Weil, M. M. (1995b). Computer availability, computer experience and technophobia among public school teachers. Computers in Human Behavior, 11(1), 9-31.

Rosenberg, M. S. (2005). The file-drawer problem revisited: a general weighted method for calculating fail-safe numbers in meta-analysis. Evolution, 59(2), 464-468.

Rosenthal, R. (1979). The file drawer problem and tolerance for null results. Psychological Bulletin, 86(3), 638.

Rotherham, A. J., & Willingham, D. T. (2010). “21st-Century” skills: Not new, but a worthy challenge. American Educator, Spring 2010, 17-20.

Rothstein, H. R., Sutton, A. J., & Borenstein, M. (2006). Publication bias in meta-analysis: Prevention, assessment and adjustments: John Wiley & Sons.

Ruben, B. D. (1999). Simulations, games, and experience-based learning: The quest for a new paradigm for teaching and learning. Simulation & Gaming, 30(4), 498-505.

Saettler, P. (1990). Early forerunners: before 1900. The Evolution of American Educational Technology, 23-52.

Salkind, N. J. (Ed.). (2010). Encyclopedia of research design (Vol. 1). Sage.

Sandholtz, J., Ringstaff, C., & Dwyer, D. (1990). Teaching in high tech environments: Classroom management revisited, first-fourth year findings. Apple Classrooms of Tomorrow Research Report Number 10. Retrieved from https://www.apple.com/euro/pdfs/acotlibrary/rpt10.pdf

Schmidt, D. A., Baran, E., Thompson, A. D., Mishra, P., Koehler, M. J., & Shin, T. S. (2009). Technological pedagogical content knowledge (TPACK): The development and validation of an assessment instrument for preservice teachers. Journal of Research on Computing in Education, 42(2), 123.

Schuh, K., & Barab, S. (2008). Philosophical perspectives. Handbook of research on educational communications and technology, 67-82.

338

Selwyn, N. (2015). Technology and education—why it’s crucial to be critical. In S. Bulfin, N. Johnson, & C. Bigum (Eds.), Critical perspectives on technology and education (pp. 245- 255). New York, NY: Palgrave Macmillan.

Shulman, L. S. (1986). Those who understand: Knowledge growth in teaching. Educational researcher, 15(2), 4-14.

Strudler, N., & Wetzel, K. (1999). Lessons from exemplary colleges of education: Factors affecting technology integration in preservice programs. Educational Technology Research and Development, 47(4), 63-81.

Tamim, R. M., Bernard, R. M., Borokhovski, E., Abrami, P. C., & Schmid, R. F. (2011). What forty years of research says about the impact of technology on learning a second-order meta-analysis and validation study. Review of educational research, 81(1), 4-28.

Teo, T. (2009). Modelling technology acceptance in education: A study of pre-service teachers. Computers & Education, 52(2), 302-312.

Teo, T., Lee, C. B., & Chai, C. S. (2008). Understanding pre‐service teachers' computer attitudes: applying and extending the technology acceptance model. Journal of Computer Assisted Learning, 24(2), 128-143.

Triandis, H. C. (1971). Attitude and attitude change: Wiley New York.

Tristán-López, A., & Ylizaliturri-Salcedo, M. A. (2014). Evaluation of ICT Competencies Handbook of research on educational communications and technology (pp. 323-336): Springer.

U.S. Congress Office of Technology Assessment. (1995). Teachers & technology: Making the connection. Washington, DC: Government Printinng Offic.

U.S. Department of Education. (1996). Getting America's students ready for the 21st century: Meeting the technology literacy challenge: a report to the nation on technology and education.

U.S. Department of Education (Office of Educational Technology). (2016). Future ready learning: Reimagining the role of technology in education (2016 national education technology plan). Retrieved from Washington, D.C.: http://tech.ed.gov/netp/

U.S. Department of Education (Office of Educational Technology). (2017). Reimagining the role of technology in education: 2017 national education technology plan update. Retrieved from Washington, D.C.: http://tech.ed.gov/netp/

339

Uman, L. S. (2011). Systematic reviews and meta-analyses. Journal of the Canadian Academy of Child and Adolescent Psychiatry, 20(1), 57.

Valentine, J. C., Pigott, T. D., & Rothstein, H. R. (2010). How many studies do you need? A primer on statistical power for meta-analysis. Journal of Educational and Behavioral Statistics, 35(2), 215-247. van Braak, J. P., & Goeman, K. (2003). Differences between general computer attitudes and perceived computer attributes: Development and validation of a scale. Psychological Reports, 92(2), 655-660. van Merriënboer, J. J. G. v., & Kirschner, P. A. (2012). Ten steps to complex learning a systematic approach to four-component instructional design (2nd ed.). New York: Routledge.

Vannatta, R. A., & Beyerbach, B. (2000). Facilitating a constructivist vision of technology integration among education faculty and preservice teachers. Journal of Research on Computing in Education, 33(2), 132-148.

Venkatesh, V., Morris, M. G., Davis, G. B., & Davis, F. D. (2003). User acceptance of information technology: Toward a unified view. MIS Quarterly, 425-478.

Viechtbauer, W. (2010). Conducting meta-analyses in R with the metafor package [Software]. Journal of Statistical Software, 36(3), 1-48. Available from http://www.jstatsoft.org/v36/i03/

Wang, K., Shu, Q., & Tu, Q. (2008). Technostress under different organizational environments: An empirical investigation. Computers in Human Behavior, 24(6), 3002-3013.

Wang, L., Ertmer, P. A., & Newby, T. J. (2004). Increasing preservice teachers’ self-efficacy beliefs for technology integration. Journal of Research on Technology in Education, 36(3), 231-250.

Willis, J., Thompson, A., & Sadera, W. (1999). Research on technology and teacher education: Current status and future directions. Educational Technology Research and Development, 47(4), 29-45.

Willis, J. M. (2015). Examining technology and teaching efficacy of preservice teacher candidates: A deliberate course design model. Current Issues in Education, 18(3).

Yavuz, S. (2005). Developing a technology attitude scale for pre-service chemistry teachers. TOJET: The Turkish Online Journal of Educational Technology, 4(1).

Zeichner, K. M. (2005). A research agenda for teacher education. In M. Cochran-Smith & K. M. Zeichner (Eds.), Studying teacher education: The Report of the AERA panel on research and teacher education (pp. 737-760). Mahwah, NJ: Lawrence Erlbaum Associates, Inc.

340

Zhao, Y., & Cziko, G. A. (2001). Teacher adoption of technology: A perceptual control theory perspective. Journal of Technology and Teacher Education, 9(1), 5-30.

341

BIOGRAPHICAL SKETCH

Matthew had a long journey on his road to becoming a doctor of philosophy, both literally and figuratively. His road began with at the University of Oregon where he received his

B.A. in German Language and Literature in 1997. Through peers he met in this program, he eventually received an English teaching position in Seoul, South Korea starting in late-2000.

While in Seoul, he taught English to learners from PreK to Adult. Most influential to his current status, he came to love teaching elementary age children at his postings after-school extension program. There he taught reading, writing, math, science, and social studies. More importantly, he felt called to pursue the next stage as an educator.

To continue his development as a teacher, he returned to his home town of Salem,

Oregon. There he enrolled the in Early Childhood and Elementary Education program at

Willamette University. The yearlong program primed him for his next steps in the classroom.

During his year home, Matthew attended the international jobs fair at the University of Northern

Iowa. This proved fruitful in yielding a position at Seoul International School. Post-graduation with his MAT degree, Matthew returned to Korea to start a new chapter.

Several important events happened for Matthew during his second stay in Korea. First, he had the amazing opportunity to teach and learn an amazing collection of students across the elementary age range. This included classroom assignments at the second and fourth grade levels and coaching opportunities (both academic and athletic) with fifth graders. Second, he became known as the “tech guy,” and as such coached peers and colleagues both at SIS and at national teaching conferences. Third, he met his lovely wife, Jungsun Leem. She guided, supported, and pushed him to achieve the next stage of a doctoral degree.

342

Matthew has been working and studying at the University of Florida since 2013. He currently teaches technology integration courses as part of UF’s ProTeach Program. He studies and researches in the field of educational technology.

343