Evaluation Across Contexts: Evaluating the Impact of Technology Integration Professional Development Partnerships Louanne Smolin Kimberly A. Lawless National Louis University University of Illinois at Chicago Abstract Professional development is a neces- ers can use technology to transform the can facilitate changes to teaching and sary component for effectively inte- teaching and learning context in a way learning in schools. They maintain that grating technology into classrooms. that will position their students for future although there is a strong perceived Unfortunately, the evaluation of tech- opportunities in the global context, pre- need for action in terms of TIPD, nology integration professional devel- paring them for the flattened world that the knowledge base derived through opment (TIPD) rarely moves beyond technology has helped to make possible. research does not guide it. In particular, participation satisfaction surveys, Through technology, teachers and stu- they advocate for more careful and more nor does it reflect the concerns of the dents can soften the boundaries between systematic approaches for document- multiple stakeholders participating in life in schools and in communities as well ing how technology integration oc- technology integration efforts. In this as between their present and future lives. curs within schools, what increases its article, the authors discuss collab- Technology has the potential to expand adoption by teachers, and the long-term orative models that hold potential for learning in ways that a traditional cur- impacts that these investments have on evaluating TIPD partnerships. They riculum cannot. teachers and students. Based on this, advocate for TIPD partners to define Yet the evaluation of technology they propose a sequential three-phase a collaborative and holistic vision integration, including professional evaluation design that includes evalua- of success that can guide the evalu- development for technology integration, tion of (1) program characteristics, (2) ation process. The authors discuss has done little to define what constitutes teacher outcomes, and (3) sustained three specific collaborative evaluation effective practices for realizing such teacher change and student achievement models, examine key issues associ- potential. So although the images of our effects. They maintain that these phases ated with implementing them, and classrooms have significantly changed should be sequential. They contend analyze how each model has the po- due to the ubiquity of technology, and that more needs to be known about the tential to strengthen and sustain pro- many teachers are incorporating tech- varieties of program structures, such as fessional development partnerships. nology in their learning environments, mentoring or train the trainers’ models, (Keywords: Technology integration, these changes have done little to truly before student and teacher outcomes can professional development, program reform education. In essence, the “more be evaluated. With this sequence, causal evaluation) things change, the more they remain the links between specific program variables same” (Sarason, 1996, p. 338). and outcomes can be substantiated. n his informative work on program This article explores the possibilities Within each of these phases, they raise evaluation, Patton maintains “that so- for collaborative evaluation of technol- key questions that should underscore Icial science has proven especially inept ogy integration professional development the design of evaluation and possible at offering solutions for the great prob- (TIPD) to transform technology practices outcomes of such an evaluation. lems of our time…. There is a pressing in schools. To achieve this transforma- Desimone (2009) proposes a need to make headway with these large tion, we need to shift and expand the re- similar, outcomes-based professional challenges and push the boundaries of search and evaluation processes we use to development evaluation model. She social innovation to make real progress” learn about how TIPD impacts classroom states that the goal when evaluat- (Patton, 2006, p. 28). One arena where practices and student learning. ing professional development is to there is great potential for pushing the move beyond participant satisfaction boundaries of social innovation forward Expanding Current Evaluation Models surveys and to systematically measure is integrating technology in schools. of Technology Integration Professional its impact on teachers and students Students are already well versed and facile Development (TIPD) in a coherent way. However, rather with using technology to shape their In their provocative review of research, than a sequential model, she advocates worlds outside of school (Jones, Johnson- “Professional Development in Inte- for a connected model in which the Yale, Perez, & Schuler, 2007). The poten- grating Technology into Teaching and evaluation of program characteristics, tial for technology to impact students in Learning,” Lawless and Pellegrino (2007) teacher practice, and student outcomes school should be realized as well. Teach- advocate that the evaluation of TIPD are recursive. Such analysis can help 92 | Journal of Digital Learning in Teacher Education | Volume 27 Number 3 Copyright © 2011, ISTE (International Society for Technology in Education), 800.336.5191 (U.S. & Canada) or 541.302.3777 (Int’l), [email protected], iste.org. All rights reserved. evaluators examine relationships among their specific classrooms, and teacher ed- teach, the quality of the participants’ what is taught in professional develop- ucators can tailor their methods courses technology-integration practices, and ment, what occurs in classrooms, and to better prepare teacher candidates for the resulting student outcomes. how it affects students. classroom-based pedagogies. Teachers are “frontline” stakehold- Both models underscore the necessity Yet often the evaluation of TIPD is ers. Through their roles, they can adjust of analyzing the impact of professional not a collaborative endeavor. Of the and fine-tune what they have learned development program activities on both stakeholders described above, we have in professional development sessions to teacher learning and student achieve- been concerned with three primary their unique contexts and students. For ment. They are noteworthy in their groups: funders, institutions of higher them, success is inextricably bound to attempts to create connected evalua- education, and teachers. From their their teaching and classroom culture. tions that link program inputs to teacher vantage points, each of these stakehold- Teachers deem professional development and student outcomes. Yet they are ers has defined their own vision for what successful if it deepens their teaching of incomplete. Both models leave out the constitutes effective TIPD, which lacks a a particular concept, helps them create importance of the number of stakehold- holistic view. instructional conditions conducive to ers involved in technology integration, For example, funders often deem student engagement, and fostering stu- as well as the impact of the stakehold- TIPD successful if it affects student learn- dent learning of content (Mumtaz, 2000). ers’ collaborative relationships on the ing. Technology integration is a costly, So it is through their direct work with meaningful integration of technology ongoing expense. External sources of students that they can incorporate what in today’s schools. They focus on what funding are necessary to maintain not they have learned within their teaching is to be evaluated rather than how. We only meaningful technology integration, practice and implement transforma- must build on these models, incorporat- but also functional technology-infused tive technology practices within their ing aspects of partnership, to deepen learning environments. Funders have classrooms. Ironically, teachers’ roles in and sustain the impact of TIPD. In our a stake in knowing whether and how evaluation are typically as limited, pas- evaluations, we must take into account their support affects teacher and student sive respondents. how to foster long-term partnerships learning. Funders want to know whether These individualistically derived where all partners benefit. the benefits of their technology fund- notions of success have resulted in A myriad of stakeholders is involved ing outweigh the costs (Kleiman, 2004; short-term changes and have hampered in the integration of technology in Lemke & Coughlin, 1998). Yet often long-term improvements in teaching schools, including funders, teachers, the outcomes of collaborative teacher and learning. Teachers have difficulty school administrators, ICT personnel, professional development are difficult sustaining the transformative practices higher education institutions, parents, to measure because they do not tend to they learn in professional develop- and community members. Working provide tangible and immediate evidence ment without ongoing support and together, these collaborators can garner (Lieberman & Grolnick, 1996). Therefore, mentorship. As such, their potential for sustained funding. They can engage in they might not reflect what funders are affecting their students’ learning, as well joint planning that benefits a continuum interested in knowing. Specifically, a as their mentorship of new teachers, is of learners from preservice teachers funder’s vision of successful TIPD is one difficult to achieve. Higher education and