December 22, 2000
Final Project Report to NSF
Andrew C. Porter, Director Barrett Caldwell and Robert Mathieu, Associate Directors
Funded by the National Science Foundation National Institute for Science Education University of Wisconsin–Madison 1025 West Johnson Street Madison, WI 53706
(608) 263-9250 (608) 262-7428 fax [email protected] http://www.nise.org
The NISE is supported by a cooperative agreement between the National Science Foundation and the University of Wisconsin–Madison (Cooperative Agreement No. RED-9452971). At UW–Madison, the National Institute for Science Education is housed in the Wisconsin Center for Education Research and is a collaborative effort of the College of Agricultural and Life Sciences, the School of Education, the College of Engineering, and the College of Letters and Science. The collaborative effort is also joined by the National Center for Improving Science Education, Washington, DC. Any opinions, findings, or conclusions are those of the authors and do not necessarily reflect the view of the supporting agencies. Final Project Report to NSF
Andrew C. Porter, Director Barrett Caldwell and Robert Mathieu, Associate Directors
National Institute for Science Education University of Wisconsin–Madison
December 22, 2000 In memory of Susan Loucks-Horsley
During the last five years, Susan Loucks-Horsley did much of her work as part of the National Institute for Science Education. Susan was not only principal investigator of NISE’s professional development program, but a member of the leadership of NISE, joining with other team leaders to guide the Institute through its program of work.
From the beginning of NISE, Susan Loucks-Horsley had a profound effect on our entire community. Committing to produce a book by the end of the first year of NSF funding seemed an ambitious—if not unattainable—goal. Susan and her coworkers did produce the book. It was a huge success and continues to have a profound effect on the thinking of people across the country as they design, deliver, and do research on professional development for teachers of science and mathematics. The book was NISE’s first “home run.” Susan’s having set and met such a goal inspired all of us in NISE to demand that level of accomplishment from ourselves.
Susan was an outstanding scholar and researcher, of course, but she was also a person committed to making a difference through her scholarship. An expert professional developer, she was a master of communication skills and, more generally, interpersonal interactions.
Although we will miss her professionalism, we will miss Susan as a friend even more. She was a wonderful, warm, and giving person. Susan made all of us better, both personally and professionally.
—Colleagues from the National Institute for Science Education Contents
Overview
Mission...... 1 Accomplishments...... 3 Strategies ...... 6 Getting the Best People...... 6 Leadership ...... 6 Self-Evaluation...... 7 Additional Funding ...... 7 Challenges ...... 8
Research Programs
Systemic Reform: Policy and Evaluation (William H. Clune and Norman L. Webb)...... 11 College Level One (Arthur B. Ellis) ...... 27 Professional Development (Susan Loucks-Horsley)...... 32 Secondary Teacher Education Project: Toward a Distributed Professional Community Model (Sharon J. Derry)...... 39
Dissemination Programs
Interacting with Professional Audiences (Senta A. Raizen) ...... 47 Communicating with Mass Audiences The Why Files Research (William Eveland & Sharon Dunwoody)...... 51
Organizational Process Programs
Cognitive Studies of Interdisciplinary Collaboration (Sharon J. Derry) ...... 59 Information Resource Coordination (Barrett S. Caldwell)...... 66 Formative Evaluation (Susan B. Millar)...... 71
Summary...... 75
Appendixes
A. NISE Staff and Collaborators...... 77
B. NISE Conferences, Forums, and Workshops ...... 89
C. NISE Publications...... 93
D. NISE-Related Publications...... 101
iii
Overview
Mission professional organizations, and universities across the country. The people and the work of the NISE share a strong and focused mission. We are While the NISE was funded by NSF through a committed to the improvement of mathematics cooperative agreement, NISE was charged with and science education. Our scope includes identifying the substantive directions for its work. practice from kindergarten through college, One of the major contributions of the NISE with attention to key transition points along proved to be its ability to identify the most the way. All of our work is directed toward the important and timely areas for science, improvement of student learning; increased mathematics, engineering, and technology student learning is the ultimate test against (SMET) education R&D. The NISE architects which we judge our success. We recognize recognized that in the EHR portfolio essentially that mathematics and science education has no investment was being made to analyze and pull disproportionately failed children from low- together large bodies of work in ways that would income and minority families and that this reveal emerging consensus. NISE not only result must be changed for moral as well as completed much needed analysis of ripe bodies of economic reasons. We see education as in the research, but interpreted that work in ways that midst of a technological revolution with great lead to productive development efforts serving the potential, but, as yet, little impact on practice. needs of practitioners (e.g., teachers, education Our work seeks to capitalize on technological administrators, professional organizations, federal advances to create new forms of educational agencies). Thus, from the very beginning, the opportunity in mathematics and science for NISE focused much of its resources on students of all backgrounds and varying conducting analytic syntheses and meta-analyses interests. The NISE has ambitious goals. We of bodies of education R&D judged to be tackle what we believe to be the toughest and important and pressing and with sufficient work to most pressing problems. Nevertheless, we are make analysis and interpretation useful. committed to a high rate of productivity and an efficient and effective coupling of our work Initially, professional development of K-12 to the improvement of practice in real settings. teachers was identified as an important and ripe target. Susan Loucks-Horsley’s team and the When the Education and Human Resources resulting book, Designing Professional Directorate of the National Science Development for Teachers of Science and Foundation created the NISE in July 1995, it Mathematics, as well as their highly successful charged its first research center to “address the dissemination efforts, proved that the choice was totality of the education enterprise, to assess good. NISE’s work led to a major its effectiveness and examine what new reconceptualization of professional development activities need to be established, what and identified a new research agenda that has been activities are no longer needed, and what new taking K-12 professional development an approaches will enhance science education.” important step forward. Obviously, the NISE shares this broad and ambitious mission with a great number of In 1995, EHR’s largest investment was in other efforts supported by NSF and other systemic reform, with programs for statewide, federal agencies, as well as by foundations, urban, and rural initiatives. The initial theory for systemic reform was more a hypothesis than an
1 established theory (Smith & O’Day, 1991). of work. We have also pursued narrower targets The NISE architects recognized the need to of opportunity for first-line empirical research. document and analyze the practices and Two examples illustrate the point: The World effects in this mushrooming and rapidly Wide Web has revolutionized communication. evolving line of reform. Again, the decision The effectiveness of NISE has in large measure proved to be wise. Bill Clune’s team profited from Web-based communication, as extrapolated, from the wealth of data about illustrated by the Web sites to support reform in practices and from the Smith and O’Day undergraduate SMET education and by the hypotheses, an empirically based theory that NISE’s own highly developed and carefully has the highly desirable property that it can be maintained Web site (http://www.nise.org). What empirically tested. At the same time, NISE NISE recognized was that (a) the Web might be a discovered that systemic reform presents new powerful tool for addressing science literacy and challenges for evaluation, challenges not (b) creating a Web site in support of science adequately met by current evaluation literacy would also offer an excellent base for procedures. After five years, Norman Webb’s research on how Web-based learning occurs and team has produced a promising book that how Web sites might be redesigned to better documents the challenges of systemic support learning. The result is the highly evaluation and proposes new methodologies. acclaimed The Why Files, created by Susan Trebach and Terry Devitt, which offers the NISE recognized the need for research-based science behind the news. Using the Web site as a tools to support the reform of first-year SMET launch pad, Sharon Dunwoody and William courses in undergraduate education. These Eveland carved out a path-breaking program of courses are the entry points for students who research on Web-based learning. will pursue majors in SMET. They are the entry points for prospective K-12 SMET Because NISE’s mission requires an ongoing teachers. They are also often the last SMET assessment of the most important SMET experiences for college students, too many of education R&D targets, it was only natural for whom become turned off to science and NISE to assess continually the productivity of its mathematics through bad experiences in their own efforts. Early work on interdisciplinary college coursework, and too many of whom problem solving, conducted by Sharon Derry, are minorities and women. Again, NISE made using NISE interdisciplinary teams as the focus of a good decision. After five years, Art Ellis’s her study, was completed in Year 3. Early team has created a family of faculty investments in formative evaluations by Susan professional development Web sites to support Millar proved so useful in the beginning years that much needed undergraduate SMET reform. this effort could be scaled back. The development of each Web site began with a research synthesis, followed by a year-long Freed up resources were redirected toward institute, staffed by experts from across the initiation of a new line of work on a Web-based country who identified promising practices approach to undergraduate teacher education in and made them available to their colleagues mathematics and science. Sharon Derry leads the through Web-based tools. Early response to effort. Many have recognized the inadequacies of the family of Web sites has been enthusiastic. current undergraduate teacher education programs to produce the kinds of teachers needed in K-12 Not all of NISE’s work has entailed big- schools; how to overcome the weaknesses in picture, analytic synthesis of an existing body undergraduate teacher education has been less
2 evident. Once again, NISE is utilizing the understanding how equity in SMET education can power of the Web to provide a whole new be achieved. Another is the role of technology in approach to undergraduate teacher education, improving the quality of SMET education and one that might ultimately solve the highly providing strategies for implementing successful problematic disconnect between innovations on a national scale. The third cross- undergraduate teacher education and the cutting theme is a commitment to dissemination. continuing professional development that All lines of the Institute’s work—professional teachers receive once leaving college and development and preservice teacher education, taking a teaching position. college-level SMET education, systemic reform— share a focus on problems of equity, profit from Figure 1 provides a schematic of NISE’s innovative uses of technology, and are committed research program in the year 2000. While the to high-quality dissemination. mission of the Institute is broad, the Institute’s program is focused and coherent. Focus is Accomplishments necessary because resources are limited, and fundamental progress requires depth of effort. NISE research on professional development in Coherence is important so that each separate science and mathematics is a good example of the line of work holds the potential for not only strength and influence of our work. Susan Loucks- achieving its goals, but informing the work of Horsley’s 1994 book, Designing Professional other lines of work. Development for Teachers of Science and Mathematics, written with Peter Hewson, Nancy The three lines of work (see Figure 1) are tied Love, and Kathy Stiles, is a top seller for Corwin together through a number of linkages to form Press. It has been put out twice in short form by a coherent focus for the Institute. Teacher the Eisenhower National Clearinghouse: once education is arguably the single most with a focus on science and once with a focus on important and most complicated piece of mathematics. Tens of thousands of copies of each successful systemic reform. At the same time, short form have been distributed to members of college-level SMET education sets standards the field. The NISE Brief that presents the that drive the K-12 SMET education system. framework for professional development has been These higher education standards are key to widely copied for use with practitioners in any successful approach to K-12 systemic workshops and courses around the country. The change. College-level SMET education also work guides the current national evaluation of the plays an essential role in defining the nature Eisenhower Program. Clearly, the field needed and quality of preservice teacher education. this synthesis and reconceptualization of While preservice teacher education and professional development in mathematics and professional development are not tightly science and gave it an enthusiastic reception. coupled, either in concept or in practice, both would surely profit from such a linkage, which Soon there will be an NISE book that does for the Institute is seeking to make. A deep evaluation of systemic reform what Susan commitment to making the work accessible to Loucks-Horsley’s book has done for professional the Institute’s target audiences permeates all development. The book by Norman Webb and activity. others not only synthesizes the field, it pushes the field forward by reconceptualizing alignment, by Three themes cross cut all of the work of the making clear that the methodologies of program Institute. One is a deep concern for evaluation are not
3 4 adequate for coping with the complexities of The NISE is equally committed to making its systemic reform, and by offering alternatives. work influential. An unusually large percentage of Also within the year, NISE’s book on a theory resources is invested in connecting NISE’s work of systemic reform by William Clune and with its target audiences. One especially effective others will be completed. The real power of mechanism is the NISE forums. Under Senta Clune’s theory lies with his ingenious Raizen and Ted Britton’s leadership, the NISE operationalization of the variables. Empirical offered its first annual forum in its initial year of tests of validity can be conducted, and he has funding. The format was to select a rapidly conducted the first. For systemic reform to developing area of NISE work, convene 300 or so remain useful, clarity and empirical tests are leading practitioners and researchers in that area, essential. and engage them in dialogue that results in a written statement capturing the current wisdom of NISE’s College Level One team, under Arthur the field. These invitational forums proved so Ellis’s leadership, is among the few groups in successful that others, including NSF, the country conducting serious research on increasingly looked to NISE to conduct forums on how to improve the quality of learning, topics they identified or to advise them on how to teaching, and assessment in undergraduate run similar events of their own. The forum SMET education. Our early work on small- strategy is now widely available through a Corwin group cooperative learning began with a meta- Press book, Designing Successful Meetings and analysis to see whether a research base existed Conferences in Education: Planning, for this often-promoted pedagogical strategy. Implementing, and Evaluating (2000). We identified a large positive effect that has caught the attention of college teachers of NISE has proven effective in producing highly science and higher education administrators visible and influential work. In five short years, across the country. Since then, we have NISE (a) built working partnerships with states, extended the work to address classroom districts, scientific and education organizations, assessments, resulting in an exciting new Web federal agencies, and foundations; (b) completed site where SMET faculty can access powerful eight books; (c) published over 100 journal alternative assessment strategies for use in articles, research monographs, and briefs their instruction (www.wcer.wisc.edu/nise/ (including two articles in the most recent issue of CL1). Current work is analyzing the impact of the premier research journal in education, the technology on student learning in Review of Educational Research, and a special undergraduate SMET education. Our research issue of Teachers College Record); (d) produced on undergraduate education and the resulting eleven Web sites including the award-winning Web-based tools have been key to NISE’s The Why Files (see http://whyfiles.news. successful involvement of scientists. wisc.edu/), which gives the science behind the news, now institutionalized in the UW–Madison As the above examples demonstrate, NISE is Graduate School budget; and (e) conducted five committed to research that steps back from the innovative and highly successful forums (and two field and takes a big-picture view, pulling additional national meetings at the request of together what is known in new ways, NSF). sometimes leading to reconceptualizations, and always pointing to promising directions for practice and for future research.
5 Strategies Horsley, who was splitting her time between WestEd in Tucson, Arizona, and the National What accounts for NISE’s tremendous Research Council in Washington, DC. For success, despite enormous challenges? During leadership of our efforts in communication and the course of five years of work, the following dissemination, we went to Senta Raizen, forming strategies have proven especially successful a partnership with her National Center for for the NISE. Improving Science Education in Washington, DC.
Getting the Best People At the suggestion of EHR/NSF, we vigorously initiated a robust program of research fellows. Typically, education research centers place Over the course of five years, 43 fellows joined unnecessary boundaries around where they NISE teams through varying arrangements and for look for scholars to involve in the work. An varying periods of time. The fellows mechanism early decision to commit NISE to allowed the NISE to recruit the most capable interdisciplinary work has paid enormous scholars and practitioners in the areas on which dividends. NISE researchers come from the NISE was focusing its efforts. Junior and senior sciences, including mathematics; the social scholars, practitioners, and members of the private sciences; and education. Of the 95 individuals sector have enhanced the quality of discourse and associated with the NISE leadership, 42 maximized the impact of NISE work. At the same percent came from education, 33 percent from time, NISE has provided opportunity and SMET fields, and 15 percent from social visibility to promising young scholars. sciences. Leadership NISE’s aggressive commitment to the strategy of interdisciplinary work can be seen in the Leadership is a key ingredient to the success of following examples. When we created The any organization, and NISE is no exception. Why Files, we went to the director of the Codirectors and associate directors have been University’s Office of News and Public used to provide complementary perspectives at the Affairs, Susan Trebach, to lead that effort. top leadership level. From day one, there has been When we saw the need for research on how parity between scientists and educators in NISE individuals learn via the Web, we went to leadership. Initially, Denice Denton, an engineer, Sharon Dunwoody, professor of journalism. and Andy Porter, an education psychologist, were When we needed policy research on systemic codirectors. In the fifth year, Porter, as director, reform, we went to William Clune, professor was joined by associate directors Robert Mathieu, of law. When we sought to launch efforts to an astronomer, and Barrett Caldwell, an engineer. reform undergraduate SMET education, we In the intervening years, Terrence Millar, a went to Arthur Ellis, professor of chemistry. mathematician, served as codirector. The stability When we sought formative evaluation, we provided by Porter and project manager Paula went to anthropologist Susan Millar. White, and the new perspectives and new ideas contributed through changes from Denton to Neither has NISE limited itself geographically Millar to Mathieu and Caldwell, served to keep to the boundaries of the University of the NISE on course, with continuing renewal of Wisconsin-Madison. When we launched our ideas. program of research on professional development, we went to Susan Loucks-
6 The most powerful leadership for the NISE National Advisory Board meetings are expensive, comes from the Team Leaders Team. NISE with members coming from across the nation. early on was organized into research teams, NISE created an advisory team, essentially a local one for each area of work. Each team is led by advisory board, comprised primarily of a senior scholar who makes NISE work a top individuals in the Madison area, most of whom priority. These leaders convene as the Team were professors at the University of Wisconsin- Leaders Team on a monthly basis. All Madison. This group proved to be especially evaluations of past progress and analyses to useful in reviewing the quality and quantity of determine new directions are conducted by the each team’s productivity and in reviewing the Team Leaders Team. Rich substantive Institute’s draft plans for new areas of work to discussions of each team’s work at the team pursue. The local advisory board, which met leaders’ table provide opportunities to learn quarterly, required no travel costs, and all from each other and to think about the work at members volunteered their time. the Institute level. EHR/NSF made a third-year review part of the Self-Evaluation initial cooperative agreement. This, too, proved to be a valuable tool for criticism and reflection. In NISE made a major investment in formative preparation for NSF’s third-year review, NISE evaluation, led by Susan Millar, an used both its national and local advisory boards to anthropologist. The work of Millar and her review the productivity of each team and to help team proved extremely valuable to the the NISE develop plans for where the work should strength and well being of the Institute in its go in the final two years. The investment in early years. In the first year, the formative formative evaluation was decreased; the work on evaluation identified some teams that were not interdisciplinary problem solving was completed; functioning as well as expected, due to team and work on undergraduate teacher education was leaders not having sufficient time assigned to begun. The third-year review also led to the activity. At the beginning of year 2, time combining the systemic policy and systemic assignments were increased, and in one case a evaluation teams into one integrated effort. For its team leader was replaced. The result was part, the NSF completed both a site visit and a dramatic improvements in productivity. Early reverse site visit for the third-year review. problems in patterns of communication were detected and addressed. Additional Funding
The NISE used two advisory boards as EHR/NSF set the budget for NISE at $2 million mechanisms for critical review, sometimes per year for each of five years. The plan for level leading to redirection of efforts. A 17-person funding proved to be a mixed blessing. Any National Advisory Board consisted of a rich productive institute needs progressively more and balanced mix of scientists, education financial support as it becomes established and researchers, education practitioners, and matures. Level funding, in that sense, inhibits representatives from business, industry, effective progress. Recognizing that level funding government, and foundations (see listing of would soon become a problem, NISE leadership membership in Appendix A). The National put in place three strategies. Advisory Board met each fall in years 2-5; each meeting focused on assessment of a First, NISE attempted to manage its NSF funding, specific aspect of NISE work. spending less in the early years so that, through
7 carryover, more funds would be available in among research, development, and practice as the later years. This was an error in judgment multidirectional and interactive; each is dependent as NSF did not permit carryover from year 4 upon insights and knowledge from the other two. to year 5 of the Institute. NISE’s second Our commitment to improving student learning in strategy was more successful, moving real settings has brought us to a strategy of activities off its budget and onto the budgets producing useful and effective tools to place in the of other organizations where ongoing activity hands of practitioners; the World Wide Web has could be sustained. One successful been an especially effective medium in that implementation of this strategy was The Why regard. We make a large and intense investment in Files. In year 3, the Graduate School at the dissemination, using multiple and complementary UW–Madison took over the effort; the strategies. Our innovative approaches to Graduate School will continue The Why Files dissemination include Web-based tools, special for the indefinite future. Another successful relationships with collaborating professional implementation of this strategy concerns the organizations, and our unique annual forums, Web sites to support reform of undergraduate together with more traditional approaches of SMET education. In particular, the Web site books, journal articles, research monographs, supporting the improvement of classroom occasional papers, and workshop series. We assessment has been spun off with continued produce timely, widely circulated briefs on NSF support to the University of New pressing topics. Mexico, under the leadership of NISE Fellow Michael Zeilik. Making the Institute work requires a powerful and robust infrastructure. Over the course of the last Yet a third strategy NISE used to deal with five years, our management structure has level funding was to secure additional support developed into an efficient and effective force for from other sources. For example, in year 5, in keeping work on time and of the highest quality— addition to the $2 million from EHR/NSF, and for producing a multitude of products. We NISE received $2 million from a variety of have evolved into a productive and unique other sources. Not only has this strategy been community of scholars, with ongoing cross- especially effective in solving the level- fertilization among teams and with our target funding problem, but it positioned the NISE audiences. This community of scholarship serves for continuing beyond its fifth year, even as a hotbed for identifying new initiatives and though EHR/NSF decided not to extend ideas. The Institute seeks to continually reinvent institute-level funding through the mechanism itself by (a) spinning off successful developed of a recompetition. products so that they can take on a life of their own through other support, (b) bringing to a close As an institute, NISE has evolved a unique lines of work that have fulfilled their intentions, and productive approach to pursuing work. and (c) initiating new lines that hold greatest We collaborate in interdisciplinary teams with promise for accomplishing our mission. a rich mix of scientists, education researchers, and educational practitioners. We employ a Challenges vigorous fellows program to bring a diverse group of both junior and senior scholars, from The NISE overcame a number of serious across the nation and around the world, to join challenges during its initial five years. our interdisciplinary teams and to give our Being the first center-level activity of the EHR work a national base. We see the interplay and funded through a cooperative agreement,
8 NISE had to negotiate with EHR the worked closely and productively with the Division procedures for working together. Initially, all of Undergraduate Education. The Professional of the EHR divisions contributed to NISE Development Team worked closely and funding. Understandably, each wanted to have productively with the Division of Elementary, a say in what work the NISE would undertake. Secondary, and Informal Education. Finally, Coming to agreement about what work would because of the quality of NISE work, because of be done in a timely fashion proved to be NISE’s increasing visibility in the field, and challenging. Even after EHR funding was because of NISE’s good relationships with the consolidated in the Research, Evaluation, and other divisions within EHR, NISE established a Communication Division (REC), frequent positive relationship with then EHR Director changes in leadership at REC made building a Luther Williams. functional relationship difficult. The final challenge to NISE was EHR’s decision Two strengths developed over time in the not to recompete the Institute for an additional relationships of NISE to NSF. First, we had five years of funding. A center is a difficult and the same project officer throughout the five- complicated organization to create and bring to year period, Larry Suter. A knowledgeable productivity. Five years of support is simply too education researcher himself, and an effective, short a period for such a fragile but valuable dedicated federal official, Suter sustained a capacity. An additional five years would positive, useful relationship with NISE. He definitely have been a wise decision on EHR’s understood the importance of NISE work and partandwouldhaveresultedinmaximum became an advocate for the Institute at the utilization of the initial five-year investment. Even EHR leadership table. Second, NISE without NSF institutional support, the NISE has developed excellent working relationships decided to continue as an institute. At this point, it with other divisions within EHR. For example, is too early to tell whether we will be successful, the College Level One Team, reforming although funding for at least one additional year is undergraduate SMET education, in hand at this writing.
9 10 Research Programs
Systemic Reform: Policy and Evaluation
The objective of the Systemic Reform Team, Policy Analysis of Systemic Reform codirected by William Clune and Norman Webb, is to make a significant contribution to I. Policy Team Mission the study of systemic reform, including how knowledge about it can be strengthened in the The Policy Analysis of Systemic Reform future. The present combined team was component of the Systemic Reform Team seeks to formed by consolidating the efforts of two generate useful knowledge about implementing previously independent research teams, Policy and improving the effectiveness of systemic Analysis of Systemic Reform (PASR) and reform in mathematics and science education. The Strategies for Evaluating Systemic Reform Policy Team focuses on meta-analyses and (SESR), and has produced a new body of syntheses of existing knowledge about systemic integrated research. Through the work of team reform in SMET education, including research, members, outside experts, conferences, evaluation, and the wisdom of the practitioner. forums, and fellows, we have generated a knowledge base that has been presented in II. Accomplishments joint activities and research papers and will soon culminate in production of two books, Convening of Scholars and Practitioners one on the theory of systemic reform and the other on the challenges to evaluating systemic The early research conducted by the Policy Team reform in mathematics and science. We have culminated in a “summer seminar” held July 25- successfully built on our core NSF work to 26, 1996, in Madison, attended by a broad group obtain outside funding for studying systemic of reformers, policy analysts, evaluators, and reform and building district capacity in the experts in the content standards and their subject Milwaukee Public Schools (funded by the matters. William Clune (1999) wrote a research Joyce Foundation, the Helen Bader monograph synthesizing the problems identified Foundation, and the Spencer Foundation) and as central in the seminar and suggesting how the for studying the impact of the Statewide future research agenda could be shaped to deliver Systemic Initiatives (funded by the National the most useful knowledge about reform. Clune's Science Foundation). These projects monograph deals with two general issues: (1) how complement the research we have done with to think about systemic reform as an effort to core NSF support by creating a synergy produce substantial gains in student achievement; between confronting and solving very (2) how to design the research to produce practical challenges while also being reflective knowledge that is useful in improving systemic and theory-driven in analyzing systemic reform in mathematics and science education. reform as a strategy for large-scale change. Michael Knapp (1997) wrote a research monograph on the implementation of systemic reform; Michael Kirst and Robin Bird (1997) on the politics of standards; and several authors (policy analysts, mathematicians and scientists) wrote papers on the national standards themselves
11 (Clune, Haimo, Roitman, Romberg, Wright, & The goal of the Forum was to generate and share Wright, 1997). The set of papers on the knowledge of systemic reform, including the role national standards resulted in a special issue of of curriculum, the role of teaching and learning, Teachers College Record (1998, Fall). Senta and the role of evaluation. To reach this goal, Raizen (1997) prepared an NISE Occasional panelists from around the nation shared their Paper on standards for science education that expertise in these various areas with participants, provides the current rationale and historical who themselves brought varied expertise to background for science standards, discusses structured conversations and networking sessions. the various meanings and interpretations The Forum resulted in a set of papers identifying associated with the standards, and addresses important issues and a synthesis paper of the think the current status of science curriculum pieces that the participants submitted around the frameworks including commonalities and central issues (Clune, Millar, Raizen, Webb, variations among them. In addition, the NISE Bowcock, Britton, Gunter, & Mesquita, 1997). Occasional Paper prepared by Michael Kirst, Robin Bird, and Senta Raizen (1997) uses Development of a Theory of Systemic Reform historical and recent concerns about mathematics and science content standards to The Policy Team conducted a telephone survey of demonstrate conflict and tensions that the staff of all SSIs funded by NSF. An interview surround the process of setting standards. protocol on the history of each initiative and While subject matter specialists are an lessons learned for policy was developed, and 48 important component of any resolution, this telephone interviews were conducted (involving Occasional Paper stresses the multiple 22 states, 21 principal investigators, 22 program perspectives that must be considered. directors, and 5 others). Documents from each state were collected (for example, proposals, Four “systemic seminars” were convened in annual reports, program evaluation reviews). Madison for the purpose of gathering more Based on the interviews and documents, information relevant to the research agenda summaries of individual SSIs were prepared. from written research and discussion with Analysis of these summaries was included in a NISE and NSF personnel. The systemic symposium conducted by the Policy Team at the seminars brought together small groups of 1999 annual meeting of the American Educational reformers, scholars, and evaluators with in- Research Association. Comprehensive analysis depth knowledge of reform. The small groups, will be included in the book on the theory of which included people from Statewide and systemic reform. Urban Systemic Initiatives (SSIs and USIs) as well as the NSF evaluation personnel, Members of the Policy Team met regularly to addressed such issues as the proper criteria for discuss the results of the survey and develop a evaluation and what works and does not work theory of systemic reform that could be tested by in systemic reform. This phase of developing the accumulating data. Initial formulations of the the research agenda yielded a new set of theory and examples of how it might be used to refined links between process and outcomes in guide analysis of data on SSIs were presented at systemic reform that guided further research. an NISE Brown Bag Lecture. The theory that emerged from these activities has provided the The theme of the 1997 NISE Forum was framework for a book on systemic reform, Theory “Research on Systemic Reform: What Have and Practice of Systemic Reform of Mathematics We Learned? What Do We Need to Know?” and Science Education. The theory presented in
12 the book will contribute to the knowledge of have studied change in large education systems; reformers, policymakers, researchers, the insights and work of fellows; an practitioners, and educational administrators interdisciplinary team of scientists, about the design, implementation, and effects mathematicians, and educators; literature on of systemic reform and represents the only systems and measuring change in education; and attempt to date to concretely specify the full our own research. Our work continues into the range of substantive and strategic factors that future by applying an embedded research model drive successful systemic reform. An NISE to build capacity in Milwaukee Public Schools Research Monograph is currently available on (funded by a second two-year grant from the systemic reform theory (Clune, 1999). Joyce Foundation from September 2000 through August 2002); developing models for analyzing Evaluation of Systemic Reform student achievement in large education systems (funded by a one-year grant from NSF from May I. Evaluation Team Mission 2000 through April 2001); and analyzing the impact of systemic reform on states (funded by a At its outset the Strategies for Evaluating three-year grant from NSF from January 1999 Systemic Reform Team developed an overall through December 2001). sequence of research and development. It set a general progression for its work over the five II. Accomplishments years beginning with (1) identifying the major issues and questions about evaluation of Convening an Interdisciplinary Team of Scholars systemic reform followed by (2) outlining and Practitioners specific strategies and models for evaluating systemic reform and then finally (3) putting A working group of scholars and practitioners met into practice what we learned. Over the five over two years to launch the study of evaluation years our work has followed this general of systemic reform. This group included direction and has become more focused with Christopher Anderson, professor of astronomy; the joining of the evaluation and policy teams. Vicki Bier, professor of industrial What we did not anticipate in 1995 at the engineering/applied mathematics; Steven beginning of this effort was the nature or the Bauman, professor of mathematics; Thomas magnitude of our current efforts to advance Carpenter, professor of mathematics education; the study of large education systems. Donald Chambers, former state supervisor of mathematics education; Susan Millar, In parallel and in cooperation with the Policy anthropologist and evaluator; Senta Raizen, Team, the overall mission of the Evaluation science educator and evaluator; Thomas Team is to advance knowledge on evaluating Romberg, professor of mathematics education; large systems. The Evaluation Team has Pat Rossman, elementary science teacher and striven to expand the existing knowledge base 1994 Wisconsin Teacher of the Year; John Witte, on systemic evaluation. We have done so by professor of political science; and John Wright, accumulating, synthesizing, and producing professor of chemistry. This group met bimonthly knowledge about systemic evaluation along to help identify the major issues that should be with developing models for doing evaluation addressed in evaluating systemic reform, the of large systems. Our work has drawn from questions that should be addressed by an the practitioners who have engaged in evaluation of systemic reform, and the profound evaluating systemic reform; scholars who changes in student learning and important
13 knowledge in SMET. Members of the working One year later, on March 13-14, 1997, over 50 group actively participated in two conferences people attended a conference in Madison on that included discussion of papers and small systemic evaluation. Attendees included SSI and working groups to identify important design USI evaluators, working group members, NSF features of evaluations of systemic reform. staff, other evaluators, and other NISE staff. The The working group was very valuable in conference successfully met its goals: strengthen informing the early thinking about evaluation the intellectual qualities of the research of systemic reform. Members of the group monographs and papers of the evaluation working identified knowledge necessary for success in group and its fellows, further the networking higher education, qualities of education not among those engaged in doing systemic necessarily assessed by standard measures of evaluations, and disseminate NISE work to others. achievement, equity issues in a male- Besides hearing presentations from four dominated field such as engineering, the evaluators of SSIs and John Witte, a UW– difficulty in identifying prerequisite SMET Madison political scientist, the participants knowledge in a range of higher education provided feedback to NISE Fellows who were courses, and the importance of using value preparing papers (Bruckerhoff, Kahle, Lee, and added analysis of standardized norm- Ridgway) and gave their thinking on designing referenced tests. systemic evaluations. The participants gave cogent advice on questions about outcomes, Networking and Conferences education systems, evaluation, issues related to design, and characteristics of evaluation designs. Networking among evaluators of NSF’s SSIs In addition to the two conferences, on two was encouraged by convening two conferences occasions SSI evaluators and other researchers and holding other small focus meetings. were invited to a small focus meeting to discuss Twenty-six people from across the nation their work and insights. Daniel Heck and Norman attended the first conference held in Madison Webb (2000) wrote a monograph synthesizing on January 4-5, 1996. Four papers were information learned from these conferences, prepared for this conference to direct the meetings, and the evaluations of SSIs. NISE discussion. Kathy Comfort, then of the Fellow Ridgway contributed to advancing American Association for the Advancement of thinking about systemic reform by synthesizing Science, prepared a paper titled “Student information from a number of fields. His Outcomes in Science.” Jim Ridgway, monograph on modeling of systems and University of Lancaster, England, and Hugh macrosystemic change (Ridgway, 1998b) drew on Burkhardt, Shell Centre of Mathematical the sciences, learning theory, epidemiology, and Education, England, prepared two joint papers, ecology to distinguish among different system one entitled “Student Outcomes in Evaluating models and indicate ways for building the System Change” and the other entitled “System evidence base for evaluating systemic reform. Alignment and System Change.” Andrew Jackman, Mt. Hood Community College, Alignment Criteria Studies Oregon, wrote a paper on assessment and educational reform that raised many questions In close cooperation with the Council of Chief about education reform, particularly as it State School Officers (CCSSO) and through a pertains to community colleges. grant to CCSSO from the National Science Foundation, Webb completed an alignment study of state standards in mathematics and science and
14 the state large-scale assessments in each of number of conferences (American Educational four states. This analysis was based on Research Association; CCSSO Large-Scale Webb’s NISE monograph on alignment Assessment Conference; Rand Conference on (1997a). An NISE brief on alignment was Assessment; National Center for Research on published in January, 1997. At the end of June Evaluation Standards and Students Testing 1998, 15 reviewers attended a four-day (CRESST); Abo Akademi University, Vasa, institute in Madison and applied four Finland; and the National Council of Teachers of alignment criteria identified in the NISE Mathematics). He also served as a consultant, monograph. An important purpose for this advised, or conducted training on alignment for institute and the analysis of the data was to the Department of Education (Title I), Indiana refine the process for judging the alignment Mathematics Initiative, Achieve, Inc., and the between standards and assessments. The American Association for the Advancement of coding by reviewers was tabulated and Science (AAAS). summarized. The report on each state describes the degree of alignment between Book on Evaluation of Systemic Reform standards and the corresponding assessments on the criteria of categorical congruence, Two NISE Fellows, Jeanne Rose Century depth-of-knowledge consistency, range-of- (Education Development Center, Newton, knowledge correspondence, and balance of Massachusetts) and Norma Dávila (University of representation (Webb, 1999c, d, e, and f). Puerto Rico), and Evaluation Team staff Webb, Heck, and Osthoff have completed 13 of 19 The summary report presents the major chapters of a book, Evaluation of Systemic Reform findings and the recommendations for in Mathematics and Science. The work on this improving the coding and analysis process book paralleled and interacted with work done by (Webb, 1999a). Based on the analyses the Policy Team on its book, Theory and Practice performed, clear differences among the states of Systemic Reform of Mathematics and Science. were evident along with common issues faced The evaluation book will serve as both a practical by all. A high percentage of standards and and a theoretical guide. Systemic reform is still a assessments across the four states failed to theory waiting for an accumulation of evidence achieve depth-of-knowledge consistency. In that will make the theory credible. The book general, too many items were below the details an approach to evaluating systemwide depth-of-knowledge level of the education reform at a state, district, or school corresponding objectives for there to be level. Central to this approach, and the centerpiece alignment. One benefit found for doing an of the book, is studying and measuring an analysis using the specific criteria is that education system’s progress on the basis of nine specific feedback was provided to states on attributes. These attributes cluster into three what could be done to improve alignment. The groups that are distinguished by how the attributes criteria and the underlying structure of the relate to systemic reform and to improving student analysis proved to be viable for detecting the outcomes within the system. A prospectus has degree of alignment and how alignment can be been submitted to two publishers for their improved. consideration. One publisher expressed interest, but has not issued a firm commitment. Based on his work on alignment through NISE and CCSSO, Webb made presentations at a
15 Synthesis on Evaluation of Systemic Reform Forum, includes papers from all of the presenters (Webb, 1999b). On February 1 and 2, 1999, the Fourth Annual NISE Forum was held. This forum was Criteria for Evaluating Education Information devoted entirely to evaluation of systemic Networks reform. Over 260 people attended from a wide range of organizations including professional Information systems are arising as an essential organizations, universities, school districts, component of systemic reform. Reform itself and states, government agencies, corporations, and the evaluation of improvement in student foundations. The Forum was organized into achievement are greatly hampered by insufficient five sessions including the overview of the data, fragmented data bases, and low knowledge conference given by Porter and Webb from of how to use data to make important decisions NISE and John Hunt from NSF; a panel on necessary to improve student achievement. One of understanding evaluation of systemic reform the key findings from our studies is that states, (chaired by Bernice Anderson of NSF and districts, and schools do not have data in a form including Juanita Clay Chambers [Detroit that can be used to monitor progress toward Public Schools], Daniel Heck [University of systemic reform. In 1998, the Evaluation Team Illinois], Zoe Barley [Western Michigan began a discussion group focusing on InfoNets, University], and Iris Weiss [Horizon information networks for education systems. The Research]); a panel on models and approaches multidisciplinary group included Susan Daffinrud, to evaluation of systemic reform (chaired by master’s degree in both mathematics and Larry Suter of NSF and including Patrick computer science; Christopher Thorn, director of Shields [Stanford Research International], WCER technical services; Barrett Caldwell, Jeanne Rose Century [Education Development professor of industrial engineering; Jeffery Center], Norma Dávila [University of Puerto Watson, graduate student in industrial Rico], and Mark St. John [Inverness engineering; Susan Zeyher, master’s degree in Research]); and a panel on findings about library science; and Jeffery Choppin, former high systemic reform from evaluations and research school teacher and graduate student in (chaired by Julio Lopez-Ferrao of NSF and mathematics education. including Jane Butler Kahle [Miami University], Daryl Chubin [NSF], Robert We inventoried existing products and activities Meyer [University of Wisconsin–Madison], that are being advanced for processing and and Clune [NISE]). After each panel, meeting the multiple functions for an education participants were divided into groups of 8 to information system. On June 28-29, 1999, 18 10 for breakout sessions of focused discussion people attended a conference to identify current on the topic of the proceeding panel. Ernest electronic network systems and to help plan what House (University of Colorado) and Cora role NISE should serve. Attendees included Marrett (University of Massachusetts- electronic system developers, a researcher in Amherst) gave their syntheses of the technologies, computer systems consultants, conference considering the presentations and district technology staff, an educational small group discussions. Marshall Smith, U.S. psychologist, and NISE staff. Based on the Department of Education, gave summary outcome of this meeting and other discussion, we remarks. Evaluation of Systemic Reform in began to develop a framework that could be used Mathematics and Science: Synthesis and to evaluate and analyze electronic information proceedings of the Fourth Annual NISE systems used in education. Thorn (2000) wrote a
16 paper on knowledge management for of Systemic Educational Reform. Members of the education information systems. Watson and writing team who participated included Zeyher (2000) wrote a paper on education LeMahieu, Heck, Century, Dávila, and Webb. system information needs and systemic Barley served as the discussant. reform. Accomplishment 8. Directorate of Education and The Joyce Foundation made a two-year grant, Human Resources (NSF) Special Emphasis Panel August, 1999, through July, 2001, to study on Evaluation. electronic information systems in schools in general and the trial of Quality School The Special Emphasis Panel on Evaluation Portfolio in particular. We are conducting this provided technical assistance and study in six schools in Milwaukee in recommendations on the full range of activities coordination with our study to develop supported under the evaluation program of the analytic capacity in the district. NSF Education and Human Resources Directorate (EHR). Dissemination of Information on Evaluation of Systemic Reform EHR’s highly visible programs needed the support of ongoing formative evaluation for their At the American Evaluation Association continuous improvement. In addition, many annual meeting in San Diego, November 6, external audiences, including the United States 1997, a panel on the role of evaluation in Congress, wanted to know what EHR’s programs education systemic reform was organized by were accomplishing in relationship to their stated Webb based on the NISE conference held in purposes. The Division of Research, Evaluation, March 1997. Panelists included Lloyd Bond and Communication (REC) provides leadership (University of North Carolina, Greensboro), for EHR program evaluation. Zoe Barley (Western Michigan University), and Patrick Shields (SRI International). At the Joint Accomplishments of the Evaluation and American Educational Research Association Policy Analysis of Systemic Reform Teams annual meeting in San Diego, April 16, 1998, Webb and NISE Fellows presented a Research on Equity symposium on evaluating systemic initiatives. NISE Fellows who presented were Ridgway, A joint accomplishment of the Systemic Reform Kahle, and Lee. Joan Herman, Teams is a set of papers, discussions, and a forum CRESST/UCLA, was the discussant. At the concerning equity in student achievement. Two American Evaluation Association annual research monographs were written, by William meeting, November 7, 1998, in Chicago, Tate (1997, mathematics) and Alberto Rodriguez members of the systemic evaluation writing (1997, science), exploring differences in student team gave a session on issues in evaluating achievement on a variety of measures and data student achievement in the context of systemic sets disaggregated by race, ethnicity, and gender. reform. Presenters were Webb, Dávila, Heck, The papers showed various kinds of differences and Osthoff with Barley providing reactions. and trends across genders and ethnic groups— At the American Educational Research European, Asian, African, Latino, and other. One Association annual meeting, April 23, 1999, in finding, for example, was of small differences Montreal, members of the writing team gave a between the genders on tests of science, but a session entitled Beyond Tradition: The Realm substantial difference in favor of males on the
17 mathematics portion of the SAT. This location. Okhee Lee prepared an NISE research mathematics/gender difference remained monograph on equity in content standards (Lee, constant across ethnic groups, which 1998), which resulted in a Review of Educational themselves differed substantially from each Research publication other in average scores. Another finding was (1999). mixed evidence for the proposition that similar course taking will produce similar The 2000 NISE Forum on Equity Issues in results. Such course taking narrows but does Mathematics and Science covered several not eliminate achievement gaps and does not important aspects of equity, including what we occur with sufficient frequency to allow know about effective methods to address equity comparisons for many students. Discussion of issues and what research is needed. Panelists these papers was far ranging and indicated a represented a wide range of expertise in equity possible need for new insights and approaches issues from across the nation. Discussion and toward equity. However, the value of networking opportunities aided the Forum in longitudinal data on outcomes disaggregated achieving its overall goal: to draw together leaders by ethnicity, gender, and subject matter was in the field and stimulate intellectually rich evident. Systemic reforms that do not collect conversations to develop a better understanding of and analyze such data are ignoring an diversity and equity issues in mathematics and important tool. science education. In the papers, presentations, and small group sessions, participants stressed the Jane Butler Kahle, who served as NISE importance of learning how to make diversity an Fellow on the Policy Team, prepared a asset rather than a problem to overcome by research monograph (1998b) and an NISE thinking of students as individuals, not as broad Brief (1998a) on reaching equity in systemic categories, by training teachers to be sensitive to reform. Both the brief and the monograph equity and diversity issues, by using strong provide analysis of educational equity in models of disseminating information, by aligning science and mathematics and, using a research curriculum, instruction, and assessment, and by paradigm, build a practical model that is conducting cross-cultural research. A synthesis of applicable for districts, schools, and the think pieces from the discussion sessions is classrooms around the country. Key available (Britton, Raizen, Kaser, & Porter, 2000). indicators, applicable across many types of educational systems, are enrollment and The Study of Systemic Reform in Milwaukee achievement in eighth-grade mathematics, Public Schools (SSR-MPS) quality of the content and instruction of science and mathematics courses, narrowing In April 1998, the Joyce Foundation agreed to of achievement gaps among subgroups of fund this project for an initial period of two years students, and changes in teaching practices to at about $1 million per year. Additional funding align them with the mathematics and science for this project was obtained from the Helen standards. Sharon Lynch, who also served as Bader Foundation, Inc., of Milwaukee. A second NISE Fellow, published a book on Equity and two-year Joyce grant has been awarded. Science Education Reform (2000) in which she uses both research-based studies and Clune and Webb codirect this project. The classroom experiences of teachers and ultimate goal of the research is understanding how students to analyze the complexities of equity systemic policy can fulfill its promise as a concerns across ethnicity, class, gender, and powerful method of sustaining widespread school
18 improvement and major gains in student development, and stimulating school achievement over the long run in the improvement. Milwaukee Public Schools (MPS). To reach that goal, the project has three principal • Research and technical assistance on how the purposes: proposed system of embedded classroom assessments in high school can be optimally 1. generate useful knowledge and successful in meeting district and school recommendations for policy in the district; goals. 2. allow impartial observers, funding agencies, and system managers to understand the system This project will both build capacity in the district and its performance at a deeper level; and and produce written reports and publications for a 3. impart analytical capacity to the district so national research audience. that SSR-MPS can be phased out, or assume a reduced role, after a period of years. Over the first 2+ years of funding, the project made great progress. Research is being conducted in six areas: policy and governance; alignment of - Focused the entire project more closely on curriculum standards, student assessments, student achievement, acquired a new project and classroom instruction; capacity of data liaison at a higher level in the district (Deputy systems and analytic expertise to deliver Superintendent Jacqueline Patterson), and relevant information about student and system obtained a letter of endorsement for the work of performance; school improvement; the project from the new Superintendent. professional development; and students with exceptional educational needs. - Became a regular and influential participant in advising the Department of Research and Research is currently being conducted in the Assessment and other departments on the new projects listed below. system of assessment and accountability currently being designed and implemented in the district. • Technical assistance and quantitative analysis on implementing the value-added - Placed a team member on the district’s data system of standardized testing currently warehouse development team. His charge is to under consideration by the School Board. work on translating district-level and school-level analytical models for the Team. • Technical assistance and analytical research on how individual schools in - Worked with two middle schools on how to Milwaukee can best acquire data from the report achievement data for accountability. district data system,andanalyze these and other data, to best meet the needs of - Worked with the Audit Office of the Board of schools. Governors on how to analyze and report adequate school performance. • Research on the extent to which the middle school proficiencies have been successful - Held a one-day workshop with MPS leadership as a means of raising standards for all and consultants from Minneapolis on assessment students, focusing professional and accountability.
19 - Produced and presented the first value-added Downer, Ron Jetty, Sarah Mason, Robert Meyer, analysis of student achievement scores in Susan Millar, Jennifer O’Day, Eric Osthoff, Andy MPS. Porter, Senta Raizen, Alberto Rodriguez, William Tate, Christopher Thorn, Paula White, and John - Gained access to and/or set up on our Wright. computers longitudinal and other data on student achievement in middle schools in four Fellows: Charles Bruckerhoff, Hugh Burkhardt, subjects. Judith Burry-Stock, Jeanne Rose Century, Kathy Comfort, Norma Dávila, Andrew Jackman, Jane - Obtained middle school proficiency data for Butler Kahle, Okhee Lee, Paul LeMahieu, Sharon all students in spring 2000 and consulted on Lynch, James Ridgway, and Uri Treisman. the database and reporting for these data. Collaborators: Christopher Anderson, Steven The project research staff also presented a Bauman, Vicki Bier, Thomas Carpenter, Donald symposium at the annual American Chambers, Deborah Tepper Haimo, Michael Educational Research Association meeting in Kirst, Michael Knapp, Judy Roitman, Tom April 2000. The session was entitled Romberg, Pat Rossman, Richard Rossmiller, Interdisciplinary Action Research on Systemic Bassam Shakhashiri, and John Witte. Reform in an Urban District. The session featured the following presentations and NISE Publications of the Systemic Reform related papers: Team
Embedded Research on Systemic Reform and Britton, E., Raizen, S., Kaser, J., & Porter, A. the Design Experiment: Similarities and (2000). Beyond description of the problems: Dissimilarities by William H. Clune Directions for research on diversity and equity issues in K-12 mathematics and science Embedded Research in Practice: A Study of education. Washington, DC: National Center for Systemic Reform in Milwaukee Public Schools Improving Science Education. by Norman L. Webb Clune, W. H. (1999). Toward a theory of systemic Looking for School Improvement in reform: The case of nine NSF statewide systemic Milwaukee: An Example of Embedded initiatives (Research Monograph No. 16). Research by Jennifer O’Day Madison: University of Wisconsin–Madison, National Institute for Science Education. Including All Students in Accountability Assessment by Jeffery P. Braden Clune, W. H., Haimo, D. T., Roitman, J., Romberg,T.,Wright,J.C.,&Wright,C.S. Value-added Measures of Student (1997). Commentaries on mathematics and Achievement by Robert Meyer science standards (Occasional Paper No. 3). Madison: University of Wisconsin–Madison, National Institute for Science Education. III. Staff, Fellows, and Collaborators Clune, W. H., Millar, S. B., Raizen, S. A., Webb, Staff: William Clune and Norman Webb N. L., Bowcock, D. C., Britton, E. D., Gunter, R. (Team Leaders), Jeffery Braden, Janice L., & Mesquita, R. (1997). Research on systemic
20 reform: What have we learned? What do we University of Wisconsin–Madison, National need to know? Synthesis of the Second Annual Institute for Science Education. NISE Forum (Workshop Report No. 4, Vol. 1). Madison: University of Wisconsin– Kirst, M., Bird, R., & Raizen, S. A. (1997). Madison, National Institute for Science Tensions between mathematics and science Education. content standards and local politics (Occasional Paper No. 5). Madison: University of Wisconsin– Clune, W. H., Millar, S. B., Raizen, S. A., Madison, National Institute for Science Webb, N. L., Bowcock, D. C., Britton, E. D., Education. Gunter, R. L., & Mesquita, R. (1997). Research on systemic reform: What have we Knapp, M. S. (1997). Between systemic reforms learned? What do we need to know? Synthesis and the mathematics and science classroom: The of the Second Annual NISE Forum (Workshop dynamics of innovation, implementation, and Report No. 4, Vol. 2). Madison: University of professional learning (Research Monograph No. Wisconsin–Madison, National Institute for 1). Madison: University of Wisconsin–Madison, Science Education. National Institute for Science Education.
Heck, D. J., & Webb, N. L. (2000). Purposes Lee, O. (1998). Current conceptions of science and issues of systemic evaluation in education achievement in major reform documents and (Research Monograph No. 20). Madison: implications for equity and assessment (Research University of Wisconsin–Madison, National Monograph No. 12). Madison: University of Institute for Science Education. Wisconsin–Madison, National Institute for Science Education. House, E. R. (1997). Implementing evaluation findings in government agencies (Occasional Raizen, S. A. (1997). Standards for science Paper No. 2). Madison: University of education (Occasional Paper No. 1). Madison: Wisconsin–Madison, National Institute for University of Wisconsin–Madison, National Science Education. Institute for Science Education.
Kahle, J. B. (1998a). Measuring progress Ridgway, J. (1998a). From barrier to lever: toward equity in science and mathematics Revising roles for assessment in mathematics education (NISE Brief Vol. 2, No. 3). education (NISE Brief Vol. 2, No. 1). Madison: Madison: University of Wisconsin–Madison, University of Wisconsin–Madison, National National Institute for Science Education. Institute for Science Education.
Kahle, J. B. (1998b). Reaching equity in Ridgway, J. (1998b). The modeling of systems and systemic reform: How do we assess progress macro-systemic change: Lessons for evaluation and problems (Research Monograph No. 9). from epidemiology and ecology (Research Madison: University of Wisconsin–Madison, Monograph No. 8). Madison: University of National Institute for Science Education. Wisconsin–Madison, National Institute for Science Education. Kirst, M. S., & Bird, R. L. (1997). The politics of developing and maintaining mathematics Rodriguez, A. J. (1997). Counting the runners and science curriculum content standards who don’t have shoes: Trends in student (Research Monograph No. 2). Madison: achievement in science by socioeconomic status
21 and gender within ethnic groups (Research Wisconsin–Madison, National Institute for Monograph No. 3). Madison: University of Science Education. Wisconsin–Madison, National Institute for Science Education. Webb, N. L. (1999a). Alignment of science and mathematics standards and assessments in four Rodriguez, A. J. (1999). Equity through states (Research Monograph No. 18). Madison: systemic reform: The case of whole-school University of Wisconsin–Madison, National mathematics and science restructuring in Institute for Science Education. Puerto Rico (Research Monograph No. 14). Madison: University of Wisconsin–Madison, Webb, N. L. (1999b). Evaluation of systemic National Institute for Science Education. reform in mathematics and science: Synthesis and proceedings of the Fourth Annual NISE Rodriguez, A. J. (2000). A cross-case analysis Forum (Workshop Report No. 8). Madison: of the Puerto Rico Statewide and the Miami- University of Wisconsin–Madison, National Dade Urban Systemic Initiatives: Promising Institute for Science Education. examples of equity in systemic reform (Research Monograph No. 21). Madison: Webb, N. L. (1999c). State A: Alignment between University of Wisconsin–Madison, National standards and assessments in science for grades 3 Institute for Science Education. and 8 and mathematics for grades 3 and 6 (Project paper for Council of Chief State School Tate, W. (1997). Race, SES, gender, and Officers and National Institute for Science language proficiency trends in mathematics Education). Madison: University of Wisconsin, achievement: An update (Research National Institute for Science Education. Monograph No. 4). Madison: University of Wisconsin–Madison, National Institute for Webb, N. L. (1999d). State B: Alignment between Science Education. standards and assessments in mathematics for grades 4, 8,and 10 (Project paper for Council of Watson, J., & Zeyher, S. (2000). Education Chief State School Officers and National Institute system information needs and systemic reform for Science Education). Madison: University of Madison: University of Wisconsin–Madison, Wisconsin, National Institute for Science National Institute for Science Education. Education. Manuscript in preparation. Webb, N. L. (1999e). State C: Alignment between Webb, N. L. (1997a). Criteria for alignment of standards and assessments in science for grades 4 expectations and assessments in mathematics and 8 and mathematics for grades 4 and 8 and science education (Research Monograph (Project paper for Council of Chief State School No. 6). Madison: University of Wisconsin– Officers and National Institute for Science Madison, National Institute for Science Education). Madison: University of Wisconsin, Education. National Institute for Science Education.
Webb, N. L. (1997b, January). Determining Webb, N. L. (1999f). State D: Alignment between alignment of expectations and assessments in standards and assessments in science for grades mathematics and science education (NISE 3, 7 and 10 and mathematics for grades 4 and 8 Brief Vol. 1, No. 2). Madison: University of (Project paper for Council of Chief State School Officers and National Institute for Science
22 Education). Madison: University of Century, J. R. (2000). Sustainability. In N. L. Wisconsin, National Institute for Science Webb, J. R. Century, N. Dávila, D. Heck, & E. Education. Osthoff (Eds.), Evaluation of systemic reform in mathematics and science. (Wisconsin Center for Webb, N. L., & Heck, D. J. (1996). Education Research, University of Wisconsin– Evaluation strategies project: A report of the Madison). Manuscript in preparation. evaluation questions conference of the evaluation strategies project (Workshop Clune, W. H. (1998). The national standards in Report No. 2). Madison: University of math and science: Developing Wisconsin–Madison, National Institute for consensus, unresolved issues, and unfinished Science Education. business. Teachers College Record, 100(1), 6-7.
NISE-Related Publications of the Systemic Clune, W. H. (1998, Fall). The “standards wars” Reform Team in perspective. Teachers College Record, 100(1), 144-149. Boone, W. J., & Kahle, J. B. (1998). Student perceptions of instruction, peer interest, and Clune,W.H.,Osthoff,E.J.,&White,P.A. adult support for middle school science: (2000). Theory and practice of systemic reform of Differences by race and gender. Journal of mathematics and science education. (Wisconsin Women and Minorities in Science and Center for Education Research, University of Engineering, 4(4), 333-340. Wisconsin–Madison). Manuscript in preparation.
Century, J. R. (2000). Capacity. In N. L. Clune, W. H., Porter, A. C., & Raizen, S. A. Webb, J. R. Century, N. Dávila, D. Heck, & (1999, September 29). Systemic reform: What is E. Osthoff (Eds.), Evaluation of systemic it? How do we know? Education Week, 19(5), 31. reform in mathematics and science (Wisconsin Center for Education Research, Dávila, N. (2000). Design of evaluation of University of Wisconsin–Madison). systemicreform.InN.L.Webb,J.R.Century,N. Manuscript in preparation. Dávila,D.Heck,&E.Osthoff(Eds.),Evaluation of systemic reform in mathematics and science. Century, J. R. (2000). Equity. In N. L. Webb, (Wisconsin Center for Education Research, J.R.Century,N.Dávila,D.Heck,&E. University of Wisconsin–Madison). Manuscript in Osthoff (Eds.), Evaluation of systemic reform preparation. in mathematics and science. (Wisconsin Center for Education Research, University of Dávila, N. (2000). Instruction. In N. L. Webb, J. Wisconsin–Madison). Manuscript in R. Century, N. Dávila, D. Heck, & E. Osthoff preparation. (Eds.), Evaluation of systemic reform in mathematics and science. (Wisconsin Center for Century, J. R. (2000). Role of evaluator. In N. Education Research, University of Wisconsin– L. Webb, J. R. Century, N. Dávila, D. Heck, Madison). Manuscript in preparation. & E. Osthoff (Eds.), Evaluation of systemic reform in mathematics and science. Dávila, N. (2000). Linkages. In N. L. Webb, J. R. (Wisconsin Center for Education Research, Century, N. Dávila, D. Heck, & E. Osthoff (Eds.), University of Wisconsin–Madison). Evaluation of systemic reform in mathematics and Manuscript in preparation. science. (Wisconsin Center for Education
23 Research, University of Wisconsin–Madison). Kahle, J. B., & Rogg, S. R. (1997). Assessing Manuscript in preparation. systemic change: Ohio's Statewide Systemic Initiative, Discovery. Systemic Initiatives, 1(3), 10- Gamoran, A., Porter, A. C., Smithson, J., & 11. Available: http://www.ehr.nsf.gov/ehr/esr White, P. A. (1997). Upgrading high school /sysinit/news.htm mathematics instruction: Improving learning opportunities for low-achieving, low-income Kirst, M. W., & Bird, R. L. (1997). The politics of youth. Educational Evaluation and Policy developing and sustaining mathematics and Analysis, 19(4), 325-338. science curriculum content standards. Advances in Educational Administration, 5, 107-132. Haimo, D. T. (1998, Fall). Are the NCTM standards suitable for systemic adoption? Knapp, M. S. (1997, Summer). Between systemic Teachers College Record, 100(1), 45-65. reforms and the mathematics and science classroom: The dynamics of innovation, Heck, D. J. (1998). Evaluating equity in implementation, and professional learning. statewide systemic initiatives: Asking the right Review of Educational Research, 67(2), 227-266. questions. Journal of Women and Minorities in Science and Engineering, 4(2,3), 161-181. Lee, O. (1999). Equity implications based on the conceptions of science achievement in major Heck, D. (2000). Purposes and questions for reform documents. Review of Educational evaluationofsystemicreform.InN.L.Webb, Research, 69(1), 83-115. J.R.Century,N.Dávila,D.Heck,&E. Osthoff (Eds.), Evaluation of systemic reform Lee, O., & Fradd, S. H. (1988). Science for all, in mathematics and science. (Wisconsin including students from non-English language Center for Education Research, University of backgrounds. Educational Researcher, 27(4), 12- Wisconsin–Madison). Manuscript in 21. preparation. Lee, O., & Paik, S. (2000). Conceptions of Heck, D. (2000). Saturation. In N. L. Webb, science achievement in major reform documents. J.R.Century,N.Dávila,D.Heck,&E. School Science and Mathematics, 100(1), 16-26. Osthoff, (Eds.), Evaluation of systemic reform in mathematics and science. (Wisconsin Lynch, S. (2000). Equity and science education Center for Education Research, University of reform. Mahway, NJ: Lawrence Erlbaum. Wisconsin–Madison). Manuscript in preparation. Osthoff, E. (2000). Incentives. In N. L. Webb, J. R. Century, N. Dávila, D. Heck, & E. Osthoff Kahle, J. B. (1997). Systemic reform: (Eds.), Evaluation of systemic reform in Challenges and changes. Science Educator, mathematics and science. (Wisconsin Center for 6(1), 1-6. Education Research, University of Wisconsin– Madison). Manuscript in preparation. Kahle, J. B. (1998). Equitable systemic reform in science and mathematics: Assessing Paik, S., & Lee, O. (1998). Analysis of the progress. Journal of Women and Minorities in conceptions of science achievement in major Science and Engineering, 4(2,3), 91-112. reform documents in the United States and Korea.
24 Journal of the Korean Association for Roitman, J. (1998). A mathematician looks at Research in Science Education, 18(4), 571- national standards. Teachers College Record, 587. 100(1), 22-44.
Paik, S., & Lee, O. (in press). Science Romberg, T. A. (1998). Comments: NCTM's achievement: Synthesis of current conceptions curriculum and evaluation standards. Teachers in major reform documents in the United College Record, 100(1), 8-21. States and Korea. Elementary Science Education (South Korea). Tate, W. F. (1997). Race-ethnicity, SES, gender, and language proficiency trends in mathematics Porter, A. C. (1994). National standards and achievement: An update. Journal for Research in school improvement in the 1990s: Issues and Mathematics Education, 28(6), 652-679. promise. American Journal of Education, 102(4), 421-449. Thorn, C. A. (2000). Knowledge management for educational information systems: What is the state Porter, A. C. (1995). The uses and misuses of of the field? ((Wisconsin Center for Education opportunity-to-learn standards. Educational Research, University of Wisconsin–Madison). Researcher, 24(1), 21-27. Manuscript in preparation.
Porter, A. C. (1998a). Dilemmas in assessing Webb, N. L. (1998a). Conditional equity metrics academic achievement. In N. M. Lambert & as tools for evaluating equity in schools and B. L. McCombs, How students learn: education systems. Journal of Women and Reforming schools through learner-centered Minorities in Science and Engineering, 4(2,3), education (pp. 339-350). Washington, DC: 141-160. American Psychological Association. Webb, N. L. (1998b). Judging alignment of Porter, A. C. (1998b). The effects of assessments and expectations in a mathematics upgrading policies on high school education system. Paper presented at a conference mathematics and science. In D. Ravitch (Ed.), on the quality aspects of mathematics and science Brookings papers on education policy (pp. education held at the Abo Akademi University, 123-172). Washington, DC: Brookings Vasa, Finland. Institution Press. Webb, N. L. (1998c). Thoughts on assessment in Raizen, S. (1998). Standards for science the mathematics classroom. In G. W. Bright & J. education. Teachers College Record, 100(1), N. Joyner (Eds.), Classroom assessment in 66-121. mathematics: Views from a National Science Foundation working conference (pp. 101-114). Rodriguez, A. J. (1998). Busting open the Lanham, MD: University Press of America. meritocracy myth: Rethinking equity and student achievement in science education. Webb, N. L. (1999a). Alignment of mathematics Journal of Women and Minorities in Science tests and mathematics reform. Mathematicians and Engineering, 4(2,3), 195-216. and Education Reform Forum Newsletter, 11(3). University of Illinois at Chicago.
25 Webb, N. L. (1999b). Judging alignment of Webb, N. L. (1999). Tools for understanding assessments and expectations in a systemic reform: Logic-of-action and theory-of- mathematics education system. In O. change models. Paper presented at theAmerican Björkqvist (Ed.), Quality aspects of Educational Research Association annual mathematics and science education. Reports meeting, Montreal, Quebec, Canada. from the Faculty of Education, No. 5. Vasa, Finland: Åbo Akademi University. Webb,N.L.,Century,J.R.,Dávila,N.,Heck,D., & Osthoff, E. (2000). Evaluation of systemic Webb, N. L. (1999c). Professional reform in mathematics and science. Madison, WI: development for improving assessment in Wisconsin Center for Education Research, mathematics and science. Paper presented at University of Wisconsin. Manuscript in the conference Weaving Together the Strands preparation of Systemic Reform: Professional Development, Assessment, and Research, Webb, N. L., & Smithson, J. (1999). Alignment Pretoria, South Africa, cosponsored by the between standards and assessments in South Africa National Research Foundation mathematics for grade 8 in one state. Paper and the U.S. National Science Foundation, presented at the American Educational Research October 27-29. Association annual meeting, Montreal, Quebec, Canada.
26 College Level One
I. Team Mission synthesis, including Web sites (all available at http://www.wcer.wisc.edu/nise/cl1), and The objective of the NISE College Level One disseminated these products to the college SMET (CL-1) Team is to enhance the quality of instructional community. The ultimate goal of the introductory postsecondary SMET courses, CL-1 Team has been to develop a comprehensive, reflecting their role as curriculum “pressure integrated Web-based resource of effective points” in shaping student career trajectories, teaching practices. The CL-1 Innovations in influencing science literacy, and promoting SMET Education Web site provides a launch pad equity. The National Science Foundation for reform-ready instructors, a support base for (NSF) published the report Shaping the campus and national professional development Future: New Expectations for Undergraduate centers and workshop providers, and a reference Education in Science, Mathematics, site for the education and research directorates of Engineering, and Technology that summarized the National Science Foundation. practices and innovative approaches for teaching postsecondary SMET courses and II. Accomplishments provided recommendations that could lead to improved outcomes in these courses. The Institute on Collaborative Learning Shaping the Future report helped the CL-1 Team align its program with NSF and In consultation with colleagues at the National stakeholder aspirations for CL-1 SMET Science Foundation, the CL-1 team investigated courses. Our efforts focused on conducting the scholarly basis for the use of cooperative research that identifies and validates effective learning in introductory college SMET courses. A practices in these courses and on meta-analysis convincingly demonstrated that disseminating this information to key using small group learning methods in SMET stakeholder audiences. We identified and courses, rather than traditional methods, produce validated practices for dissemination through significant gains in student achievement, annual CL-1 Institutes. Each CL-1 Institute persistence, and attitude. The effect size of 0.5 focused on a topic of critical importance to that was found is highly significant. This study, college SMET instruction to bridge the large “Effects of Small-Group Learning on gulf between the knowledge resulting from Undergraduates in Science, Mathematics, education research and the knowledge base Engineering and Technology: A Meta-Analysis,” and practice of college SMET instructors. The was published in Review of Educational Research topical expertise was generated by a select (Springer, Stanne, & Donovan, 1999) and group of NISE Fellows, representing both represented the first such analysis spanning SMET instructors and education researchers; SMET disciplines and the spectrum of the CL-1 Team provided the ongoing postsecondary institutions. A summary article of infrastructure. Major themes addressed by the this research for chemistry audiences was CL-1 Institutes were cooperative learning, published in the Journal of Chemical Education classroom assessment of student learning, and (Bowen, 2000). the appropriate use of technology in student learning. Each CL-1 Institute synthesized Given the importance of the findings from the what is known in higher education SMET meta-analysis, the CL-1 Team implemented an research, prepared products based on this aggressive strategy for bringing collaborative
27 learning methods to the attention of college development of a Web site, the Field-tested SMET instructors, advisors, and Learning Assessment Guide (FLAG). administrators through the development of a Web site. The heart of the Collaborative The theme of the 1998 NISE Forum was Learning Web site, developed under the “Indicators of Success in Postsecondary SMET leadership of Anthony Jacob and Ann Education: Shapes of the Future.” The Forum Burgess, is an instructor-friendly tutorial on engagedabroadcross-sectionofcollegeSMET collaborative learning techniques. In addition, stakeholders in a dialog regarding the importance the site provides personal stories and advice of assessment. A proceedings volume was from SMET instructors who have adopted this prepared by Susan Millar and is available from pedagogical approach; the research foundation NISE. As background scholarship for the for collaborative learning (including the meta- Institute’s work, a book on assessment research, analysis report); an annotated bibliography on Assessing Science Understanding. A Human collaborative learning that is available through Constructivist View, was co-edited by NISE NISE; a perspective article in Journal of Fellow Joel Mintzes and published by Academic College Science Teaching, “Small-Group Press. The volume presents critical analyses from Instruction in Science, Mathematics, national experts on various assessment tools. Engineering and Technology: A Discipline Status Report and a Teaching Agenda for the Part of the vision underpinning the Institute on Future” (Cooper & Robinson, 1998); and a Assessment of Student Learning was that the compilation of frequently asked questions. Institute’s products would help reform-ready Constructive suggestions that improved the practitioners to think of assessment beyond the site were obtained from the Communicating traditional awarding of grades. Many methods of with Mass Audiences (CMA) Team and from assessment—including portfolios, concept maps, national experts who attended a workshop and interviews—are being introduced into college hosted by CL-1. To advertise the Web site, SMET courses to better characterize student Kate Fahl developed a brochure that has been learning. A practitioner-oriented resource distributed to thousands of college SMET developed by the Institute is the Field-tested instructors and administrators through Learning Assessment Guide (FLAG) Web site, disciplinary professional meetings and which can be accessed through the above URL. collaborations with organizations such as The content of the site comprises a primer that Project Kaleidoscope. provides a rationale for integrating assessment with college SMET courses and introduces Institute on Assessment of Student Learning common terms associated with assessment; a facilitator that helps college SMET instructors As college SMET instructors begin to identify appropriate goals and align them with experiment with new forms of pedagogy and assessment methods; and a collection of so-called content, the question inevitably arises about Classroom Assessment Techniques, or CATs, and whether their students are learning more as the associated tools. The CATs were commissioned changes are implemented. A CL-1 Institute from nationally recognized leaders in the was developed to explore issues associated assessment of student learning and include with the classroom assessment of student contributions from the NSF-funded Math learning. An NISE Forum and a book Assessment Resource Service (MARS) project. developed by an NISE Fellow helped to All contributions were vetted through an Editorial launch this effort, which resulted in the Board drawn from national experts. Like the
28 Collaborative Learning Web site, the FLAG A three-day workshop was held at NSF Web site was created to serve as a core headquarters during the summer of 1999. The resource for the CL-1 SMET instructional workshop brought together leading scholars in community. Through a successor NSF grant to education and cognitive research and college former NISE Fellow Michael Zeilik, one of instructors who are pioneers in the use of the original developers of the Web site, the technology in SMET at the college level. The FLAG will continue to be updated and workshop report (TechEd 99; expanded, as befits a moving target like http://www.wcer.wisc.edu/teched99/) resulting assessment. The NISE Formative Evaluation from this meeting identified key technical, Team’s initial evaluation of the FLAG site pedagogical, political and economic issues indicates that it is meeting its objectives of associated with the introduction of technology. serving the college SMET instructional community as a central assessment resource. With support from the NSF CISE Directorate’s As was done for the cooperative learning National Partnership for Advanced Computing project, an aggressive dissemination effort Infrastructure (NPACI), the 1999-00 CL-1 was implemented, including preparation of a Institute on Learning Technology studied how widely distributed brochure and presentations SMET faculty are successfully using new at high-profile conferences, including technologies in their classrooms and laboratories. meetings of the American Educational Under the leadership of NISE Fellow Susan Research Association (AERA) meeting and Millar, the CL-1 Institute was to provide college the American Association of Higher SMET instructors with better resources for Education (AAHE). understanding why and how they should use technology-enhanced learning strategies. The Institute on Learning Technology Team (a) conducted ten case studies of exemplary technology use in a variety of SMET disciplines The introduction of technology into college and higher education environments; (b) developed SMET courses is changing the landscape a set of twenty “vignettes,” illustrating a range of associated with these courses. New technology-enhanced practices in disciplines and technologies include computers, sensors, and types of institutions not represented in the case multimedia presentation tools. Collectively, studies; and (c) gathered from a diverse sample of such technologies are reshaping college 200 SMET faculty responses to frequently asked SMET education by providing opportunities questions about the use of technology in college for distance learning, access to enormous SMET courses. Analysis of all these materials databases, and remote use of state-of-the-art revealed key factors for successful innovation and instrumentation, to cite just a few examples. adaptation of technology in support of student The CL-1 Institute on Teaching with learning. A comprehensive taxonomy of learning Technology was established to explore the technologies that informs the case study analysis impact of technology and to help college and helps shape future research in learning SMET instructors identify appropriate uses of technology was also developed. The materials technology. A Technology Education resulting from these studies have been workshop served as a springboard for this incorporated into another CL-1 Web site. In effort, and the Institute has developed a Web addition, the findings will be developed into site that is intended to serve as a resource for workshop materials for distribution by the college SMET teaching community. professional organizations.
29 III. Staff and Collaborators Collaborative Learning brochure (available Staff: Arthur B. Ellis (Team Leader), Robert through NISE central office) D. Mathieu (Director, CL-1 Institute), Briana Bright, Gina Brissenden, Aaron Brower, Ann Field-tested Learning Assessment Guide brochure Burgess, Susan Daffinrud, Samuel Donovan, (available through NISE central office) Jack Husted, Anthony Jacob, Kate Loftus Fahl, Walter Secada, and Leonard Springer. Learning Through Technology brochure (available through NISE central office) Fellows: Jean-Pierre Bayard, Lia Brillhart, Craig Bowen, Steve Ehrmann, John Jungck, Student Assessment of their Learning Gains Eileen Lewis, Flora McMartin, Joel Mintzes, brochure (available through NISE central office) Marco Molinaro, Elaine Seymour, Sheila Tobias, and Michael Zeilik. NISE Publications of the College Level One Team Collaborators: Clifford Adelman, James Cooper, Patricia Cross, Douglas Duncan, Cooper, J., & Robinson, P. (1998). Small-group Roscoe Giles, Mary Kennedy, Louise Liao, instruction: An annotated bibliography of science, Catherine Middlecamp, Susan Millar, mathematics, engineering and technology Terrence Millar, John Moore, Greg Moses, resources in higher education (Occasional Paper Jeanne Narum, James Ridgway, Pamela No. 6). Madison: University of Wisconsin– Robinson, Gloria Rogers, Karl Smith, and Madison, National Institute for Science Mary Beth Stanne. Education.
Products Millar, S. B. (Ed.). (1998). Indicators of success in postsecondary SMET education: Shapes of the Collaborative Learning Web site: future. Synthesis and proceedings of the Third http://www.wcer.wisc.edu/nise/cl1/CL/default. Annual NISE Forum (Workshop Report No. 6). asp Madison: University of Wisconsin–Madison, National Institute for Science Education. College Level One Web site: http://www.wcer.wisc.edu/nise/CL1 Springer, L., Stanne, M. E., & Donovan, S. S. (1998). Effects of small-group learning on Field-tested Learning Assessment Guide undergraduates in science, mathematics, (FLAG) Web site: engineering, and technology: A meta-analysis http://www.wcer.wisc.edu/nise/cl1/flag/default (Research Monograph No. 11). Madison: .asp University of Wisconsin-Madison, National Institute for Science Education. Learning Through Technology Web site: http://www.wcer.wisc.edu/nise/cl1/ilt/default. NISE-Related Publications of the College Level asp One Team
Student Assessment of their Learning Gains Adelman, C. (1997). Leading, concurrent, or (SALG) Web site: lagging? The knowledge content of computer http://www.wcer.wisc.edu/salgains/instructor/ science in higher education and the labor market
30 (ED Publication No. PLLI 97-8046). Mintzes, J. J., Wandersee, J. H., & Novak, J. D. Washington, DC: U.S. Department of (Eds.). (1998). Teaching science for Education. understanding: A human constructivist view.San Diego, CA: Academic Press. Adelman, C. (1998). Women and men of the engineering path: A model for analysis of Mintzes, J. J., Wandersee, J. H., & Novak, J. D. undergraduate careers (ED Publication No. (Eds.). (2000). Assessing science understanding: PLLI 98-8055).. Washington, DC: U.S. A human constructivist view. San Diego, CA: Department of Education. Academic Press.
Bowen, C. W. (2000). A quantitative literature Springer, L. (1998). Research on cooperative review of cooperative learning effects on learning in college science, mathematics, high-school and college chemistry engineering, and technology. Cooperative achievement. Journal of Chemical Education, Learning and College Teaching, 8(3), 2-4. 77(1), 116-119. Springer, L., Stanne, M. E., & Donovan, S. S. Committee on Undergraduate Science (1999). Effects of small-group learning on Education, National Research Council. (1999). undergraduates in science, mathematics, Transforming undergraduate education in engineering, and technology: A meta-analysis. science, mathematics, engineering, and Review of Educational Research, 69(1), 21-51. technology. Washington, DC: National Academy Press. Tobias, S. (1999). Some recent developments in teacher education in mathematics and science. A Cooper, J., & Robinson, P. (1998). Small review and commentary. Journal of Science group instruction in science, mathematics, Education and Technology, 8(1), 21-31. engineering, and technology: A discipline status report and a teaching agenda for the Wiese, D., Seymour, E., & Hunter, A. (1999, future. Journal of College Science Teaching, June). Report on a panel testing of the student 27(6), 383-388. assessment of their learning gains instrument by faculty using modular methods to teach Mathieu, R. (2000). Assessment tools to drive undergraduate chemistry. Boulder: University of learning: FLAG, SALG, and other proven Colorado, Bureau of Sociological Research. assessments available online. In Targeting curricular change: Reform in undergraduate Wright, J. C., & Wright, C. S. (1998). A education in science, math, engineering, and commentary on the profound changes envisioned technology (pp. 26-31). Washington, DC: by the national science standards. Teachers American Association for Higher Education. College Record, 100(1), 122-143. Wright, J. C., Millar, S. B., Kosciuk, S. A., Millar, S. (2000). Opening doors to new Penberthy, D. L., Williams, P. H., & Wampold, B. spaces: The processes of implementing E. (1998). A novel strategy for assessing the education reform. In Targeting curricular effects of curriculum reform on student change: Reform in undergraduate education competence. Journal of Chemical Education, 75, in science, math, engineering, and technology 986-992. (pp. 41-45). Washington, DC: American Association for Higher Education.
31 Professional Development
I. Team Mission II. Accomplishments
The NISE Professional Development Team The Team’s accomplishments in addressing each dedicated itself to identifying what constitutes of the seven goals are outlined below. effective professional development for science and mathematics teachers and spreading that Goal 1: To document the characteristics of knowledge to education leaders working to effective professional development for science improve science and mathematics teaching in and mathematics teachers and develop a K-12 classrooms. Through its five-year framework to guide the redesign of program of work the Team has had a professional learning opportunities for significant impact on the professional teachers. development practices in many reform initiatives, school districts, and state • Developed and disseminated NISE Brief, departments of education. It has helped staff Principles of Effective Professional developers throughout the nation to think Development for Mathematics and Science more critically about their own practices. Education (Loucks-Horsley et al.), a synthesis of the professional development standards The questions that drove the Team’s work produced by the National Council of Teachers were: of Mathematics (NCTM), the National • What is currently known about science Research Council (NRC), the National Staff and mathematics, learning, teaching, Development Council (NSDC), and the professional development, and the change National Center for Improving Science process that can improve the design and Education. effectiveness of learning opportunities for teachers? • Created the Professional Development Design • How should the specific goals of Framework. In collaboration with NISE professional development for a particular fellows Hubert Dyasi, Susan Friel, Judy set of teachers and the context in which Mumme, Cary Sneider, and Karen Worth and teacher learning opportunities are being advisors Josefina Arce, Joan Ferrini-Mundy, planned influence the design of Deborah Schifter, Vernon Sells, Mark St. professional development? John, and Iris Weiss, the Team developed and • What alternative strategies and approaches documented the process through which to professional learning exist, how are they effective professional development is used, and how effective are they in designed. The goal of developing a framework assisting teachers to learn what they need for the design of professional learning to know to effect student learning of opportunities for K-12 inservice science and science and mathematics? mathematics teachers evolved during the first • What practices help to bridge the gap year as a direct result of work with the NISE between preservice education and teacher fellows. The intent was to capture the learning professional development? of current professional development efforts to • What skills and knowledge do professional increase the knowledge base from both “craft developers need in order to provide high wisdom” and disciplined inquiry and make quality learning experiences for teachers? that information accessible to practitioners and
32 researchers alike. At team meetings, and vignettes, often drawn from the work of professional developers shared their work the NISE Fellows. The audiences for the book by describing how they had developed are, first, those in positions to design and their programs and by explaining the conduct professional development and, critical decisions they had made that second, those who fund, consume, conduct contributed to the particular forms their research on, evaluate, and develop policies for programs had taken. The outcome of these professional development. deliberations was a consensus that effective professional development • Collaborated with the Eisenhower National practice could best be characterized by Clearinghouse to disseminate findings more means of a Design Framework. The broadly. Summarized the Design Framework decisions that are made in designing and the 15 major professional development professional learning opportunities for strategies in two publications—one for science science and mathematics teachers are the educators and one for mathematics educators. product of a cyclical design process that is Over 50,000 of these books have been informed and influenced by various inputs. disseminated throughout the nation. The Team identified the inputs to professional development design that must Goal 2: To create a dialogue among be considered, including (1) contextual professional developers and other education factors such as existing policies, teacher leaders for elaborating and understanding knowledge and skills, curriculum, student what constitutes effective professional achievement levels; (2) the knowledge development and for promoting the use of the base on teaching, learning, the nature of NISE Design Framework and other findings. science and mathematics, professional development, and the change process; (3) • Engaged with leaders in professional critical issues such as how the project will development. The project engaged with reach scale, how it will develop professional development leaders around the professional culture, address equity, and NISE findings through annual and regional evaluate its results; and (4) the professional meetings and ongoing email professional learning strategy or communication. The Team conducted sessions combination of strategies (e.g., coaching, at conferences of NCTM, the National Science study groups, workshops) that will best Teachers Association (NSTA), and NSDC address the professional development each year and invited participants to try out needs and purpose. the findings and recommendations of the project and communicate with staff about their • Wrote and disseminated a book on the results. Many participants maintained contact Professional Development Design through email and follow-up at other Framework. The Team summarized meetings. The Team also conducted learning findings and recommendations in the book sessions for NSF systemic initiatives and Designing Effective Professional teacher enhancement projects, the Glenn Development for Teachers of Science and Commission, WestEd’s National Academy for Mathematics, published by Corwin Press Science and Mathematics Education in 1998. Over 11,000 copies have been Leadership, large school districts, and many sold. Each of the key sections of the regional and state service providers. Through framework is elaborated with illustrations these efforts over 3,000 leaders of professional
33 development became aware of the NISE contexts and the organizations, structures, and Professional Development Design methods that serve to facilitate their effective Framework and ideas for improving implementation and dissemination. The case learning for science and mathematics studies were committed to using the NISE teachers. The Professional Development Design Framework to analyze the results of Team has also interacted with preservice completed efforts or plan ongoing efforts and educators and professional developers documenting and analyzing their from around the country in an ongoing implementation efforts. dialogue about what is known about effective development of science and Edward Silver, Margaret Smith, and Mary mathematics teachers and the critical Kay Stein wrote retrospective case studies of issues that surround professional two sites that were part of the QUASAR development. Project, a multiyear project funded by the Ford Foundation (1990-1995) that focused on • Conducted First Annual NISE Forum, the teaching of mathematics in middle schools 1996. The Forum’s primary purposes were in large urban school districts. Cases to (1) extend and strengthen understanding examined the professional development of practices and issues in professional efforts at these two sites, using the design development, with implications for design framework as a tool for describing and of more effective programs and initiatives analyzing the professional development in mathematics and science education, and decisions that were made. The case studies (2) build a community of those seeking drew on existing data about teachers, resource ongoing interaction around professional partners from local universities, and the development practices and issues. A professional development experiences secondary purpose was to design a collected over the five years of the project. conference that modeled many of the The cases detail the curricular decisions made principles of effective professional (one site implemented a reform-oriented development. The Forum included a mathematics curriculum developed through plenary presentation of the Design funding from the NSF; the other site used a Framework, discussions of eight case curriculum to be created by the teachers and studies of professional development in unified by the concepts of ratio and science and mathematics, and discussions proportion) and the professional development of critical issues that cut across all activities used to support these curricula. professional development efforts, e.g., These cases are valuable in helping equity, leadership, scaling up. professional developers understand the importance of key factors such as the Goal 3: To develop case examples of the mentoring by teachers of their new colleagues NISE Professional Development Design while they adopted the school’s innovative Framework in action. curriculum and relationships between teachers and resource partners from local universities. • Developed four case studies of professional development practice. The NISE staff Susan Mundry and fellow Ned Team studied the ways in which the Levine wrote a case study about a staff Design Framework and its central development project funded by the NSF as a components are implemented in different Rural Systemic Initiative project involving 42
34 schools in 22 school districts in application of the Design Framework with the southwestern Montana. The project goal Team. was to implement new strategies in science and technology education in these • Published case study findings. The Team and schools through a variety of professional collaborators conducted a cross-case analysis development strategies such as teacher and identified a number of dilemmas faced by institutes, leadership training, mentor professional developers as they planned and teacher training, and direct classroom carried out their efforts. These were described support. The case considered three in three publications—in the Harvard schools: a small school (about 300 Educational Review, The High School students), a one-room school, and a school Principal Magazine, and the NISE Brief. (See serving predominantly Native American product list.) students. It focused on the question of the integrity of the curriculum as it was • Presented cases at 1998 AERA annual implemented across distance and in meeting different contexts of culture and local control. The case is valuable in providing Goal 4: To investigate and document professional developers with insights into conditions and/or programs that bridge the the difficulties imposed on effective gap between preservice and inservice professional development by distance, education. isolation, and cultural context. • Conducted a literature review and prepared a Team member Kathy Stiles wrote a case summary paper on the relationship between study about science and mathematics the preservice and inservice components of professional development in a large urban science and mathematics teachers’ school district that is culturally diverse. In professional development and the principles the district, professional development in that should guide the development of this science and mathematics was designed and relationship in the future. coordinated by six people who formed a team that worked with the district Staff • Convened collaborators with expertise in Development office to ensure that district particular approaches to connecting pre- and and program goals were met. For science inservice teacher education (e.g., in and mathematics, curricular goals included professional development schools, in novice application of science and mathematics in teacher mentor and induction programs, in the real world, competence in the use of teacher assessment). The collaborators were technology, and a diversification of chosen because they had implemented teaching strategies. Professional programs or approaches to bridge the development activities were designed to preservice-inservice gap, brought different increase teacher skills and confidence in perspectives, e.g., from colleges, schools, or the use of reform-oriented teaching state education agencies, and had disciplinary strategies and to facilitate backgrounds in mathematics and/or science. interdisciplinary, thematic instruction. The They wrote a detailed statement of the case describes the history of professional problem and the factors that contribute to the development planning, implementation in disconnect between preservice and inservice the district, and the intervention of Stiles’ and generated recommendations for
35 developing stronger, more coherent chapter and worked closely with the relationships between schools and collaborators to develop a coherent set of colleges. cases on the three professional development strategies. • NISE Fellow Barbara Spector, with Susan Mundry, Katherine Stiles, and Susan Goal 6: To study the effects of knowledge Loucks-Horsley, documented initiatives utilization strategies to improve professional and conditions that build connections development practice. between preservice and inservice education and summarized them in a • Established and convened a cadre of compendium and report prepared as NISE professional developers who are using the Research Monograph No. 17. (See product project’s research to make improvements in list.) professional learning experiences for teachers of science and mathematics. The Professional Goal 5: To deepen understanding of three Development Team documented how the different professional development cadre uses knowledge from NISE research to strategies, their impact in different make improvements in teacher learning. contexts, with different teacher learning goals, and in different combinations. • Drafted guidebook of learning activities for preparing professional developers to design • Prepared a literature review on three and provide quality teacher learning specific strategies for professional experiences. development—curriculum implementation, immersion in inquiry and • Designed and conducted professional learning problem solving, and case discussions— activities. The Professional Development team entitled “Strategies for Professional and PD cadre developed and tried out over 40 Development: What Do We Know and learning activities designed to increase Still Need to Know?” professional developers knowledge of effective professional development and • Documented three specific professional professional development design. Activities development strategies that build that were successful in meeting the desired pedagogical content knowledge in science outcomes are included in the guidebook draft. and mathematics. The Team recruited NISE collaborators to investigate three III. Staff, Fellows, and Collaborators strategies—curriculum implementation, immersion in inquiry and problem solving, Staff: Susan Loucks-Horsley (Team Leader), and case discussions—more deeply. The Peter Hewson, Nancy Love, Susan Mundry, staff, collaborators, and team fellow Katherine Stiles, Kathy Dunne, and Mary Cheryl Brown-Kovacic convened three Stenson. times to explore the purpose and outcomes of these strategies and to demonstrate how Fellows: Cheryl Brown-Kovacic, Hubert Dyasi, teachers engage in them. Collaborators Susan Friel, Uwe Hilgert, Ned Levine, Judith wrote papers that describe an intervention Mumme, Margaret Schwan Smith, Edward Silver, using the different strategies. Staff and Cary Sneider, Barbara Spector, Mary Kay Stein, Brown-Kovacic wrote an introduction and Karen Worth.
36 Collaborators: Josefina Arce, Carne Barnett, Mundry, S., Spector, B., Stiles, K., Loucks- Virginia Bastable, Cathy Carroll, Karen Horsley, S. (1999). Working toward a continuum Cerwin,KathyDiRanna,MarkDriscoll, of professional learning experiences for teachers Kathy Dunne, Joan Ferrini-Mundy, David of science and mathematics (Research Monograph Hartney, Gregg Humphrey, Barbara Miller, No. 17). Madison: University of Wisconsin– Jean Moon, Jan Phlegar, Lynn Rankin, Ann Madison, National Institute for Science Rosebery, Susan Jo Russell, Deborah Schifter, Education. Vernon Sells, Jerome Shaw, Mark St. John, Rita Starnes, Jo Topps, and Iris Weiss. NISE-Related Publications of the Professional Development Team NISE Publications of the Professional Development Team Eisenhower National Clearinghouse. (1998). Ideas that work: Mathematics professional Hewson, P., & Loucks-Horsley, S. (1997). development. Columbus, OH: Author. (Digest Professional development for science and of mathematics portion of Loucks-Horsley, mathematics education: Putting knowledge Hewson, Love, & Stiles, 1997) into action. A synopsis of the First Annual Forum of the National Institute for Science Eisenhower National Clearinghouse. (1999). Education (Workshop Report No. 3). Ideas that work: Science professional Madison: University of Wisconsin–Madison, development. Columbus, OH: Author. (Digest of National Institute for Science Education. science portion of Loucks-Horsley, Hewson, Love, & Stiles, 1997) Loucks-Horsley, S., Hewson, P., Love, N., & Stiles, K. (1998). Designing professional Horsley, D. L., & Loucks-Horsley, S. (1998). development for teachers of science and CBAM brings order to the tornado of change. mathematics. Thousand Oaks, CA: Corwin Journal of Staff Development, 19(4),17-20. Press. Loucks-Horsley, S. (1998, Spring). The role of Loucks-Horsley, S., Stiles, K., & Hewson, P. teaching and learning in systemic reform: A focus (1996). Principles of effective professional on professional development. Science Educator, development for mathematics and science 7(1), 1-6. education: A synthesis of standards (NISE Brief Vol. 1, No. 1). Madison: University of Loucks-Horsley, S. (1998, Summer). JSD forum: Wisconsin–Madison, National Institute for I have changed my emphasis. Journal of Staff Science Education. Development, 19(3), 7-8.
Mundry, S., & Loucks-Horsley, S. (1999). Loucks-Horsley, S., & Bybee, R.W. (1998). Designing professional development for Implementing the National Science Education science and mathematics teachers: Decisions Standards. The Science Teacher, 65(6), 22-25. points and dilemmas (NISE Brief Vol. 2, No. 1 ). Madison: University of Wisconsin– Mundry, S., Spector, B., Loucks-Horsley, S., & Madison, National Institute for Science Morrison, N. (1999, December 1). From novice to Education. pro: Improving science and mathematics teaching by improving teacher learning. Education Week, 19(14), 21.
37 Mundry, S., Spector, B., Loucks-Horsley, S., Stein, M. K., Smith, M. S., & Silver, E. A. (1999). & Morrison, N. (2000, January-February). The development of professional developers: From novice to pro: Improving science and Learning to assist teachers in new settings in new mathematics teaching by improving teacher ways. Harvard Educational Review, 69(3), 237- learning. NSTA Reports, 11(4), 53. 269.
Stein, M. K., & Mundry, S. (1999, October). Stiles, K. E., & Loucks-Horsley, S. (1998). Professional development for science and Professional development strategies. The Science mathematics teachers: Dilemmas of design. Teacher, 65(6), 46-49. The High School Principal Magazine, 7(2), 14-19.
38 Secondary Teacher Education Project
I. Team Mission professional fields involving application of complex domain knowledge to real-world practice Reforming teacher education is a current (e.g., medical education). national priority (NRC, 1996). This continuing project is about promoting standards-based II. Accomplishments reform in teacher education through Web- based instructional design. An important Development of STEP Web Prototype question addressed through our work is how to effectively build Web-based instruction that Our work involved design and implementation of helps teachers acquire truly useful knowledge the STEP Web (http://www.wcer.wisc.edu/step), a that supports them at work and in other real- learning environment on the World Wide Web for world contexts. Such knowledge encompasses supporting scientific instruction for teacher not only pedagogical and subject-matter education. The phrase scientific instruction refers concepts and skills that underlie teaching, but to both the design of the Web site and its subject also the tools of lifelong, self-directed matter content; both are based on current professional development. principles of learning and development drawn from cognitive psychology and other learning The phrase useful knowledge emphasizes an sciences. STEP Web and its associated ambitious goal for knowledge transfer, that instructional approaches were tried and evaluated knowledge acquired in one context (e.g., in teacher education programs at UW-Madison school, Web sites) be recalled and used in and Rutgers University in the 1999–2000 spring another context (e.g., the workplace). semester. The field trials were successful; an Knowledge transfer has been an elusive goal upgraded version of STEP Web is currently being for education; researchers, instructional used and evaluated in the UW-Madison teacher developers, and knowledgeable educators education program. In the following we describe eagerly seek ways to obtain it. Studies by the STEP site, the theoretical rationale for its cognitive psychologists and anthropologists design, and its use in teacher education. We then have repeatedly shown that nontrivial summarize the results of our field trials and other knowledge transfer is difficult to achieve (e.g., research. Salomon & Perkins, 1989). The STEP Web goals are to help future teachers The premise is that high level transfer of acquire pedagogically relevant structural professional knowledge and skills, including knowledge about the disciplinary concepts they the skills of reflective practice and lifelong are teaching; scientific knowledge about how learning, can be attained through Web-based students learn and develop within disciplines; and instructional design. By Web-based the ability to combine, adapt, and apply this instructional design we refer to both design of knowledge base to analyze and improve teaching Web environments and their communication and learning. tools, and the social structures of their use. Little is known about how to engineer A theory about how to design instruction that will effective instructional Web environments. insure transfer of learning to professional life is This work will lead to general design solutions Cognitive Flexibility Theory (CFT; e.g., Spiro, and principles that may apply to many Feltovich, Jacobson, & Coulson, 1991).
39 Appropriate domains for the application of permit navigation through the Knowledge Web CFT include professional-level education, (such as Web links among video cases and case- such as medical and teacher education. A relevant learning research and theories). The central argument of CFT is that many resulting Knowledge Web represents a instructional approaches fail because they realistically complicated, nonlinear network of represent complex subject matter in an conceptually complex relationships among cases unrealistically simplified and well-structured and ideas within a domain. manner. The most common kind of failure observed is reductive bias—the tendency to Accordingly, STEP Web is a highly oversimplify approaches and solutions to interconnected network of pages that interlink the complex problems. The remedy for learning following types of resources in a conceptually deficiencies related to domain complexity meaningful manner: requires instructional designs that afford greater cognitive flexibility. • cases—stories of student learning and development resulting from lessons in This includes the ability to represent actual classrooms—that include edited knowledge from different conceptual video of the classroom plus additional and case perspectives and then, when materials that supply information about knowledge must later be used, the context ability to construct from those different • instructional problems and projects that conceptual and case representations a make use of cases and are designed to knowledge ensemble tailored to the promote in-depth analysis for development needs of the understanding or problem- of knowledge about how to support solving situation at hand. (Spiro et al., student learning 1991, p. 24) • a network of case-related links to Web pages and other resources discussing core A decade ago the developers of CFT concepts from cognitive psychology and suggested that instruction with other learning sciences that teachers need multidimensional, nonlinear hypertext systems to know has the power to convey knowledge • access to expert case analyses complexity and promote features of cognitive • links to online discussions and human flexibility. Today this observation applies to expertise site design on the World Wide Web. A CFT • links to additional tools and resources that approach to instructional Web design proceeds teachers can use to help them adapt and as follows: (1) A problem-solving domain is implement ideas acquired from study of defined (such as application of subject-matter cases knowledge and scientific learning principles to teaching); (2) a library of cases representing Implementation and Evaluation of STEP Web real-world problems and solutions is constructed (such as video-based stories about In the 2000 spring semester, 55 teacher education actual classroom lessons); (3) a library of students were assigned to small groups of 5–7 resources for understanding and analyzing the students that studied together within a Problem cases is developed (such as Web pages about Based Learning (PBL) format. During the learning theories and educational research); semester, each student participated in two and (4) a system of connections is created to different PBL groups. Each group was assigned a
40 problem and a case to study. For example, a experts. After revision, a group report was case assigned to a group of science majors submitted as a course requirement. was “Students Get a Charge out of Static Electricity.” This case, presented on STEP The STEP implementation at UW-Madison during Web as readings, videos, and inquiry the 2000 spring semester represented a process of materials, tells the story of an actual science continuous user-centered design in which students unit in a public school taught by a popular provided feedback that was used to upgrade and teacher and representing a good case of improve STEP Web throughout the semester. traditional instruction. The problem was to Early in the semester, intensive feedback was redesign the unit and justify that redesign in obtained from a small number of students who psychological terms. volunteered to be research subjects, but on March 7 and again on April 18, all students were The work of the students was guided in class surveyed to obtain their feedback and satisfaction by a tutor—a TA who received training in ratings regarding the Web site. guiding PBL. After studying the case individually on STEP Web, students began Fifty-four of the 55 students returned surveys on their group work in class by discussing the March 7; 50 returned surveys on April 18. On case and identifying learning issues—things these dates, 48 and 46 students respectively they needed to learn more about to solve the reported using the Web site as an instructional redesign problem. Between classes, students resource for their study and PBL research. researched their learning issues, bringing Satisfaction with STEP Web as an instructional findings to their group discussions. STEP Web resource was 3.9 on March 7 and 4.1 on April 18, was made available as an optional research based on a rating scale of 1–5 (not very satisfied tool that could be used outside of class. The to very satisfied). Students’ comments initiated a links and navigational tools in STEP Web number of improvements and changes throughout guided students’ research; by exploring paths the semester. For example, the addition of a from links, students were able to obtain search engine was based on students’ requests. information about case-relevant concepts and Students’ satisfaction with STEP Web increased other resources and pursue them to a desired as the site was improved and students gained depth. Research beyond the materials in STEP experience. For example, one student who Web was also promoted, since links led to participated in three surveys commented: other library and Web resources. Some students also purchased and used optional • Feb 22: “KW (Knowledge Web) - textbooks. impressedmethisweek....Ididnot research outside of it.” (No rating The problem required about four weeks to requested) complete. A tutor guided students through • Mar 7: “I am getting better at navigating class discussions of their research, during the KW.” (Rating = 4) which time they identified positive and • Apr 18: “I am starting to appreciate the negative aspects of the instruction within the Knowledge Web.” (Rating = 5) case and proposed new instructional solutions. In the third week they posted their redesign Other representative student comments: with explanations on a Web conference site for peer evaluation and consultation with • “When I finally figured out how to use it, it experts, including scientists and educational was great.” (Rating = 5)
41 • “I like the newer KW.” (Rating = 4) individuals’ conceptual understandings of student • “Much improved!” (Rating = 4) learning and development were documented. • “Some pages that could have helped These findings suggest that the STEP instructional weren’t up.” (Rating = 4) model as a whole is viable in terms of course • “They [Web pages] were quite useful, goals. but KW needs to be more easily navigated.” (Rating = 3) Development of Video Cases for STEP • “I found the KW to be confusing in some of its explanations.” (Rating = 3) Recent work suggests that teachers improve their instruction when presented opportunities to view In sum, most students in the UW-Madison and discuss cases of classroom practice (e.g., course were pleased with the Knowledge Web Merseth, 1996; Shulman, 1992). Facilitated case by midsemester, but their comments indicated discussions are deemed an especially promising that further development and improvement is method for teacher education, since preservice needed. Based on students’ concerns, there is teachers often do not have control of classrooms need to (1) add to and improve resources on in which to try out their teaching ideas (Putnam & STEP Web, (2) improve navigation, and (3) Borko, 2000). provide instructional supports within the course to speed the process of learning how to Publications from the Third International use the site. We are currently working on Mathematics and Science Study (TIMSS; e.g., these improvements. Stigler & Hiebert, 1999), coupled with release of videotapes of 235 eighth-grade mathematics STEP Web was also used at Rutgers lessons (http://timssvideo.psych.ucla.edu) from University in a smaller course taught by an three countries, has increased enthusiasm for experienced PBL instructor. There it was using video cases of classroom practice as a basis positively rated, despite being at an early stage for case discussions. Many researchers are now in its development. Based on fourteen students designing and investigating various strategies, and a scale of 1–5 (not very satisfied to very technologies, and social structures for helping satisfied), the Web site was rated 4.6. The make video case discussions an aspect of teaching textbook used in the course, a best-selling culture (e.g., Barab et al., 2000; Frederiksen et al., educational psychology text, was rated 4.5. 1999; Lampert & Ball, 1998; Marx, Blumenfeld, Krajcik, & Soloway, 1998). For example, Barab et The evaluation study at UW-Madison al. developed a video-based Internet technology produced statistical evidence of significant that enables teachers to upload video of their growth in students’ ability and propensity to classes and remain in their classrooms while they activate and combine concepts from the go online to observe and discuss how teachers in learning sciences in the analysis of videotaped other sites are implementing state standards. Marx lessons. In addition, a microgenetic analysis of et al. (1998) describe a multimedia system called a single PBL group, composed of science CAPPS (Casebook of Project Practices) that education majors, demonstrated that scaffolds teachers as they study and learn from discussions of cases provided opportunity for video cases of classroom practice. Stigler learners to socially construct meanings, share (personal communication) and Goldman-Segal knowledge, negotiate conflicts, and integrate (personal communication) are developing Internet ideas from other teacher education courses technologies to support teachers’ group and experiences. Significant shifts in
42 discussions and learning from instructional video cases. We have made significant progress in building and testing a new conferencing tool that supports Unlike most approaches mentioned above, distributed (distance) PBL in STEP. This addition STEP requires Internet video cases that are will also provide new users with early guidance in carefully selected, edited, and designed to use of STEP Web and will therefore minimize teach a particular body of curricular some reported difficulties in learning to use the knowledge (Derry, Kim, Steinkuehler, Street, site effectively. Canty, Fassnacht, Hewson, Hmelo, & Spiro, 2000). Cases include many components: References edited video of classrooms, expert analyses, teacher commentary, and supplementary Barab, S., Moore, J., Cunningham, D., & the ILF materials. We have developed six instructional Design Team. (2000, April). The Internet video cases and associated instructional Learning Forum: A new model for online materials. These cases represent lessons in professional development. Paper presented at the middle and high school science (2 cases); high annual meeting of the American Educational school mathematics; high school foreign Research Association, New Orleans LA. language; high school English; and high school social studies. More cases are currently Frederiksen, J. (1999, November). Supporting under development, and we are seeking funds professional development and improving practice to continue and expand this work. using video case examples of classroom practices collected for that purpose. Paper presented at the Web Tools for Supporting Distributed meeting of the Board on International (distance forms) of PBL Comparative Studies in Education, National Academy of Sciences invitational meeting, Building tools to support a distributed form of Washington, DC. PBL is necessary due to the widespread problem of resources for undergraduate Hmelo, C. E., & Williams, S. M. (Eds.). (1998). education. By putting STEP PBL activities Learning through problem solving [Special issue]. online, we will avert many physical, temporal, The Journal of the Learning Sciences, 7(3&4). and human resource constraints that face STEP and greatly increase the potential Lampert, M., & Ball, D. L. (1998). Teaching, number of students served. Our strategy multimedia, and mathematics: Investigations of involves adding features to STEP Web that real practice. New York: Teachers College Press. will better prepare each individual group member to use the STEP Web site and to Marx,R.W.,Blumenfeld,P.C.,Krajcik,J.S.,& participate in group work. Also, most group Soloway, E. (1998). New technologies for teacher discussions will be removed from the professional development. Teaching and Teacher classroom to an asynchronous online Education, 14(1), 33-52. environment. This approach will permit facilitated PBL groups for distance courses National Research Council. (1996). The national and will have a number of other advantages science education standards. Washington DC: over our current STEP implementation, which National Academy Press. requires that each group meet face-to-face in class each week with a trained tutor.
43 Pea, R. (1993). Practices of distributed III. Staff and Collaborators intelligence and designs for education. In G. Salomon (Ed.), Distributed cognitions (pp. 47- Staff: Sharon J. Derry (Team Leader), Brian 87). New York: Cambridge University Press. Bietzel, Barrett Caldwell, Nicole Canty, Kate Hewson, Jong-Baeg Kim, Julia Lee, Julie Posselt, Putnam, R. T., & Borko, H. (1998). Teacher Constance Steinkuehler, John Street, and Jeffrey learning: Implications of new views of Watson. cognition. In B. J. Biddle, T. L. Good, & I. F. Goodson (Eds.), The international handbook Fellow: Marcelle Siegel of teachers and teaching. Dordrecht, The Netherlands: Kluwer. Collaborators: Paul Feltovich, Cindy Hmelo, andRandJ.Spiro. Salomon, G., & Perkins, D. N. (1989). Rocky road to transfer: Rethinking mechanisms of a Products neglected phenomenon. Educational Psychologist, 24, 113-142. Canty, N. G., & Caldwell, B. (1999, May). Information flow and Web site design: Using the Shulman, L. (1992). Toward a pedagogy of World Wide Web to support resources. Paper cases.InJ.H.Shulman(Ed.),Using case presented at and appearing in the proceedings of methods in teacher education (pp. 1-30). New the Eleventh Annual National Black Graduate York: Teachers College Press. Student Conference, Baton Rouge, LA.
Siegel, M., Derry, S., Kim, J., Steinkuehler, Derry, S. J. (2000, April). Reconceptualizing C., Street, J., Canty, N., Fassnacht, C., teacher education: Video case-based problem Hewson, K., Hmelo, C., & Spiro, R. (2000). solvingontheWorldWideWeb. Paper presented Promoting teachers’ flexible use of the at the annual meeting of the American learning sciences through case-based problem Educational Research Association, New Orleans, solving on the WWW: A theoretical design LA. approach. In B. Fishman & S. O’Connor- Divelbiss (Eds.), Proceedings of the Fourth Derry, S. J. (2000). Tutoring and knowledge International Conference of the Learning construction during problem-based learning: An Sciences (pp. 273-279). Mahwah, NJ: interaction analysis. Paper accepted for Erlbaum. presentation at the 2001 annual meeting of the National Association for Research in Science Spiro, R. J., Feltovich, P. J., Jacobson, M., & Teaching, St. Louis, MO. Coulson, R. L. (1991). Cognitive flexibility, constructivism and hypertext: Advanced Derry, S. J., & DuRussel, L. A. (1999, July). knowledge acquisition in ill-structured Assessing knowledge construction processes in domains. Educational Technology, 31(5), online learning communities. Paper presented at 24-33. and appearing in published proceedings of AIED99, Conference of the International Society Stigler, J. W., & Hiebert, J. (1999). The for Artificial Intelligence, Lemans France. teaching gap. New York: The Free Press.
44 Derry, S., Kim, J., Steinkuehler, C., Street, J., Siegel, M. (2000). But electricity isn’t static: Canty, N., Fassnacht, C., Hewson, K., Hmelo, Science discussion identification of learning C., & Spiro, R. (2000). Promoting teachers’ issues and use of resources in a problem-based flexible use of the learning sciences through learning education course. Paper accepted for case-based problem solving on the WWW: A presentation at the 2001 annual meeting of the theoretical design approach. In B. Fishman & National Association for Research in Science S. O’Connor-Divelbiss (Eds.), Proceedings of Teaching, St. Louis, MO. the Fourth International Conference of the Learning Sciences (pp. 273-279). Mahwah, Steinkuehler, C. A., & Derry, S. J. (2000). What NJ: Erlbaum. types of knowledge do you want to disqualify in the very instant of your demand, “Is it a Derry, S., Kim, J., Steinkuehler, C., Street, J., science?” A discourse analysis of online social Canty, N., Fassnacht, C., Hewson, K., Hmelo, argumentation. Paper accepted for presentation at C., & Spiro, R. (2000, October). Taking the 2001 annual meeting of the American STEPs toward infusing modeling and Education Research Association, Seattle, WA. visualization technologies into teacher education. Paper prepared for NSF-sponsored Steinkuehler, C. A., Derry, S. J., Levin, J. R., & workshop on Infusing Modeling and Kim, J.-B. (2000, April). Argumentative Visualization Technologies into Teacher reasoning in online discussion. Paper presented at Education, Washington, DC. the annual meeting of the American Educational Research Association, New Orleans, LA. Derry, S. J., & Steinkuehler, C. A. (2000). Video cases for teacher enhancement. Paper Watson, J., Caldwell B., Derry, S. J., & Canty, N. accepted for presentation at the 2001 annual (1999, September). Sociotechnical design aspects meeting of the American Education Research of educational technology. Paper presented at and Association, Seattle, WA. appearing in published proceedings of the Human Factors and Ergonomic Society Annual Meeting, Derry, S. J., Steinkuehler, C. J., & Fassnacht, Houston, TX. C. (2000, October). Web-based instruction for inclusive problem-based learning. Poster NISE-Related Publication of the Secondary presented at CILT 2000, Washington, DC. Teacher Education Project
Kim, J.-B., Derry, S. D., Steinkuehler, C. A, Derry, S. J., Gance, S., Gance, L. L., & Schlager, Street, J. P., & Watson, J. (2000, April). Web- M. (2000). Toward assessment of knowledge based online collaborative learning. Paper building practices in technology-mediated work presented at the annual meeting of the group interactions. In S. Lajoie (Ed.), Computers American Educational Research Association, as cognitive tools II: No more walls. Mahwah, NJ: New Orleans, LA. Erlbaum.
45 46 Dissemination Programs
Interacting with Professional Audiences
I. Team Mission II. Accomplishments
The IPA team reaches out to diverse The Annual NISE Forum professional audiences to encourage use of NISE products and research results, obtain The five annual NISE Forums have been very reactions to them, and elicit suggestions for successful in four ways: (1) creating awareness future research directions. The IPA Team does and a generally favorable perception of NISE and this directly through organizing the annual its work, (2) generating state-of-the-art knowledge NISE Forums, but also assists other NISE on pressing topics in the SMET community, (3) teams in planning and conducting helping shape NISE’s and individual teams’ dissemination activities. Each Forum research agendas, and (4) developing an effective convenes representatives of the many NISE process for all participants to contribute to the target audiences to share research and substance and results of large-scale SMET developments in an area of science and education meetings. Throughout the five years, mathematics education reform selected for its the NISE Forum has been fully subscribed, timeliness and salience. The IPA Team also drawing on a rich mix of some 300 participants at has recruited and is working with some 25 each Forum; it also has attracted supplemental selected professional organizations that are corporate support each year. Individuals from collaborating with the NISE more intensively. virtually every U.S. state have represented many These collaborating organizations assist in the kinds of institutions (dozens of professional work of NISE and in disseminating NISE organizations, all types of postsecondary results in ways that are tailored for reaching institutions, a variety of research and development the organizations' members. projects, and diverse types of school districts). Participants' feedback from the formal evaluation Specifically, the IPA team’s goals are questionnaire as well as anecdotal comments have been overwhelmingly positive. The feedback from • to share knowledge and information each Forum has been used to shape the succeeding generated by the NISE with key one. stakeholders; • to promote interaction and dialogue about The caliber of Forum participants has made it very the knowledge and information generated worthwhile to collect and analyze the perspectives by the NISE among key stakeholders; and offered in the Forum's small-group discussions • to encourage policymakers and and include them in the Forum reports. The practitioners to use research knowledge to experts who have been enlisted to speak also have improve science, mathematics, contributed to the reports through their prepared engineering, and technology education. papers that are circulated in advance of the Forum. The continuing value of the Forums is
47 evidenced by the publications resulting from the IPA staff and has been published by Corwin them or having been partially shaped by the Press. panel papers and discussions. For example, the Year 1 Forum discussions led to the The Forum is a good example of how the Institute commercially published book by Susan has successfully drawn together the work of its Loucks-Horsley and her PD team. Teachers teams. For the first four Forums, though the IPA College Press has expressed interest in Team coordinated all aspects of the Forum, the monographs authored by the Systemic Reform cognizant research team was responsible for the Team, one on policy analysis and one on substantive content and agendas. The research evaluation of systemic reform initiatives, with teams have worked hard not only to highlight their inputs from the Forums held in Years 2 and 4. NISE work but also to draw in leading experts The Year 3 Forum on assessment issues in from other organizations. The Year 5 Forum was postsecondary education helped launch the an all-Institute effort, since equity issues cut Web site FLAG (Field-tested Learning across all the teams’ research. The NISE Assessment Guide), an interactive site posting management regularly monitored and facilitated assessment examples designed and used by the development of each year's Forum. The SMET faculty. The Year 5 Forum on equity Formative Evaluation (FE) team has helped issues in SMET has proved important enough design the details of the Forum evaluations as well that the U.S. Department of Education as analysis of small-group discussions to feed into supported the writing and production of a the published reports. summary report, to be widely distributed. Collaborating with Other Organizations Some papers written by the Forum speakers have independently been receiving acclaim. The IPA Team initiated relationships with some Susan Loucks-Horsley's Year 1 paper was 25 organizations interested in collaborating with reproduced in Science Educator,the NISE. As a result, several organizations have newsletter of the National Science Educational formally lent their names and/or efforts to putting Leadership Association. Many requests for on one of the annual NISE Forums: American Uri Treisman's Year 2 paper led to the Association of Higher Education, Consortium for Institute plans to transform it into an NISE Policy Research in Education, Council of Chief Brief. The National Research Council State School Officers, National Council of circulated several papers from the Year 3 Teachers of Mathematics (NCTM), National Forum to participants in one of its conferences Governors’ Association, National Staff on undergraduate education. Development Council (NSDC), and National Science Teachers Association (NSTA). Also, the The effectiveness of the Forum processes to publication divisions of NCTM and NSDC generate and disseminate knowledge is formally codistributed the PD Team's book, and evidenced by the fact that NSF requested the the Eisenhower National Clearing House NISE IPA Team’s assistance in conducting produced two spin-off publications based on the two other similar conferences in SMET book, one in mathematics and one in science. education, one on teacher preparation and one Senior staff of several of the collaborating on graduate education. Moreover, a guide to organizations have provided critical reviews of designing effective meetings and conferences, draft NISE publications and attended working based on the NISE development and meetings of NISE research teams. Based on their experience with the Forums, was prepared by knowledge of NISE work, NCTM, NSDC, and
48 (particularly) NSTA have invited NISE staff 1998 Special NISE Forum. “Strengthening to make featured presentations at their annual Graduate Education in Science and Engineering: and/or regional conferences, and several Promising Practices and Strategies for organizations have asked NISE staff to confer Implementation” with them on issues of mutual interest. 1999 Fourth Annual NISE Forum. “Evaluation of Dissemination Assistance Systemic Reform in Mathematics and Science”
The most notable success of the IPA Team's 1999 Special NISE Forum. “From Preparation to efforts to work with NISE staff on Practice: NSF Teacher Education PI Workshop” dissemination activities is its work to negotiate and execute the commercial 2000 Fifth Annual NISE Forum. “Research on publication of the PD team's work by Corwin Equity Issues in SMET Education” (Report to be Press and the secondary distribution of the published and distributed in fall 2000.) book by NCTM and NSDC. This book is making far more impact than if it had been NISE Insider, quarterly electronic newsletter to self-published by NISE. For example, in its NISE Collaborating Organizations first month, the book already had become one of Corwin's top three all-time sellers. As noted 1997-1999 annual meetings of NISE earlier, the Institute looks forward to two Collaborating Organizations further externally published monographs, initially facilitated by the IPA team. Also, NISE Publications of the Interacting with Education Week accepted two articles Professional Audiences Team submitted by the Team (see NISE-Related Publications). Britton, E., Raizen, S., Kaser, J., & Porter, A. (2000). Beyond description of the problems: III. Staff Directions for research on diversity and equity issues in K-12 mathematics and science Senta Raizen (Team Leader), Edward Britton, education. Washington, DC: National Center for Mary Ann Huntley, and Susan Mundry. Improving Science Education.
Products Clune, W. H., Millar, S. B., Raizen, S. A., Webb, N. L., Bowcock, D. C., Britton, E. D., Gunter, R. 1996 First Annual NISE Forum. “Professional L., & Mesquita, R. (1997). Research on systemic Development for Mathematics and Science reform: What have we learned? What do we need Education: Putting Knowledge into Action” to know? Synthesis of the Second Annual NISE Forum (Workshop Report No. 4, Vol. 1). 1997 Second Annual NISE Forum. “Systemic Madison: University of Wisconsin–Madison, Reform: What Do We Know? What Do We National Institute for Science Education. Need to Know?” Clune, W. H., Millar, S. B., Raizen, S. A., Webb, 1998 Third Annual NISE Forum. “Indicators N. L., Bowcock, D. C., Britton, E. D., Gunter, R. of Success in Postsecondary SMET L., & Mesquita, R. (1997). Research on systemic Education: Shapes of the Future” reform: What have we learned? What do we need
49 to know? Synthesis of the Second Annual NISE NISE-Related Publications of the Interacting Forum (Workshop Report No. 4, Vol. 2). with Professional Audiences Team Madison: University of Wisconsin–Madison, National Institute for Science Education. Clune, W. H., Porter, A. C., & Raizen, S. A. (1999, September 29). Systemic reform: What is Webb, N. L. (1999). Evaluation of systemic it? How do we know? Education Week, 19(5), 31. reform in mathematics and science. Synthesis and proceedings of the Fourth Annual NISE Mundry, S., Britton, E., Raizen, S., & Loucks- Forum (Workshop Report No. 8). Madison: Horsley, S. (2000). Designing successful University of Wisconsin–Madison, National professional meetings and conferences in Institute for Science Education. education: Planning, implementation, and evaluation. Thousand Oaks, CA: Corwin Press. White, P. A. (1998). NISE Fellows program: Feedback from past Fellows (Occasional Mundry, S., Spector, B., Loucks-Horsley, S., & Paper No. 7). Madison: University of Morrison, N. (1999, December 1). From novice to Wisconsin–Madison, National Institute for pro: Improving science and mathematics teaching Science Education. by improving teacher learning. Education Week, 19(14), 21.
50 Communicating with Mass Audiences
I. Team Mission sciences. However, these and most other general science news sites tend to recycle print The general objectives of the CMA Team have news or to simply post news releases from been to promote public understanding of SMET- research organizations. Curiously, there are related concepts and issues and to understand how very few venues developing purely online the new communication channel of the World science content such as that produced by The Wide Web can be used most effectively to Why Files. communicate SMET concepts to mass audiences. During the first three years, the CMA Team’s Research Component primary effort was designing and producing The Why Files (http://whyfiles.news.wisc.edu), the Beginning in Year 2, the CMA team award winning World Wide Web science conducted evaluative research on The Why magazine created by NISE. At the end of Year 3, Files specifically and more general research funding for The Why Files site was shifted from on how science learning takes place on the NISE to the Graduate School of the University of World Wide Web. This latter work became Wisconsin-Madison. At that time, the focus of the the sole focus of the team beginning in Year 4, team became conducting research on the resulting in a change in the CMA team communication of science via the World Wide leadership. The research plans for the 4-year Web, an endeavor the team began in Year 2. period were to (1) understand the users of science information on the Web (e.g., Production Component demographics, motivations), particularly users of The Why Files; (2) understand how While traditional mass media such as newspapers, individuals use science information on the television, and radio play a critical role in Web, particularly their navigation patterns informing the public about news and issues of through Web sites; and finally, (3) understand science and technology, they are often limited in the effects of Web presentations of science their resources and ability to provide depth and information on information processing and context. And while traditional media pique public learning. Thus, our research program began curiosity about science and technology, The Why with a descriptive focus during Years 2 and 3 Files seeks to satisfy that interest by offering in- and moved into a more explanatory phase for depth, informed, and critical looks at current Years 4 and 5. topics such as Lyme disease, planets discovered beyond our solar system, and genetic engineering. II. Accomplishments to Date
Within the last two years, the media landscape has The Production Component been altered as new science news venues have emergedontheWorldWideWeb.Whileafewof The Why Files continues to expand its unique these sites seek to capitalize on popular interest in content base, producing between 45 and 50 science, many cater to specific constituencies. For new illustrated feature articles per year. In the instance, there are a number of science news past year, The Why Files has covered an array venues that provide news and information for of topics such as moving species from the audiences connected in some way to the life endangered list and the discovery of marine
51 worms that live to be an astounding 250 years of student project assistant, a valuable asset that age. Other packages covered the battle to teach has helped the production team maintain its evolution in Kansas, the world’s malaria epidemic rigorous publication schedule and improve the and the technology that underpins a proposed site in numerous ways. Inclusion of a project missile defense system. The Why Files production assistant also provides important opportunities team is now engaged in a site redesign exercise to for graduate students who can take the Web- improve navigation and accessibility for Why content production skills they learn at The Files readers. A number of those changes have Why Files to their future employers. been driven directly by the research conducted on The Why Files site by members of the CMA An important new extension of The Why Files Team. now exists in the form of an online introductory geoscience course taught through Changes in package design and approach to a the UW-Madison Department of Geology and given topic may be subtle or pronounced, but the Geophysics. Inspired by geology professor basic formula that has made The Why Files a and MacArthur Foundation Fellow Jill popular success remains. That formula involves a Banfield, and developed in collaboration with carefully crafted combination of accurate but geology professor Phil Brown, the course was humorously written text, eye-catching graphics, taught for the first time in spring 2000 to a photos, selected hot links to other Web sites, and three-figure enrollment. The department plans occasionally video and sound. That formula is to continue offering the course at least once now employed on a weekly basis to produce a every academic year. The course is based feature package that keys off of events in the entirely on Why Files content with added news. By keeping to a rigorous production material to ensure adequate coverage of basic schedule, the site has expanded considerably in ideas and principles in the geological and the past year, and the dynamic nature of the site is physical sciences. a feature that draws a loyal and growing following. Another key feature of the site is the During Years 4 and 5, The Why Files Cool Science Image gallery, which draws on continued to maintain a profile as one of the pictures and images from the front lines of most popular and critically acclaimed science- scientific research. The site also hosts an active related sites on the World Wide Web. By net forum and the occasional contest, both of providing an in-depth look at the SMET which boost the site’s drawing power. content of current affairs and news, The Why Files has developed an informational niche During Years 4 and 5, there have been several that attracts a broad audience in terms of both new developments. The most significant demographics and interest. development was the move of The Why Files production team from the National Institute for Use of the site remains steady. Statistics Science Education to the Graduate School of the gathered indicate that more than 100,000 University of Wisconsin-Madison on July 1, individual computers tap into The Why Files 1998. The move has placed The Why Files on a in any given month. People from at least 120 permanent institutional footing and enabled it to different countries have visited the site. expand its activities and collaborations to take The Why Files also continued to enjoy critical The Why Files in important new directions. success and garners awards and citations at a Moreover, it has permitted us to expand Why steady pace. Some examples include selection Files staff through the addition of a graduate as one of the “50 Hottest Sci-Tech Sites” by
52 Popular Science, a site award from SchoolZone other organizations have established direct where teachers serve as judges, a Top Site Award links to The Why Files over the course of the from Education Planet, inclusion in Web Feet: past year. An invited presentation was given The Internet Traveler’s Desk Reference, a in Salt Lake City at the first annual Science featured site award from StudyWeb, and a 1999 Education on the Internet meeting in the fall Best of the Web Award from HotSheet.com. of 1999. This meeting was open only to a Positive critical reviews of The Why Files site select, invited group of science education Web appeared in numerous publications, including the site developers from the United States and Chicago Tribune,theFlorida Times-Union,the Canada. In addition, Why Files presentations LosAngelesTimes, Discover magazine, the and informational sessions were provided at Riverside Press-Enterprise,and4kids.org,a Madison’s 1999 Future Fair and the spring syndicated feature appearing in more than 150 2000 meeting of the Wisconsin Society of newspapers in the United States and Canada. Science Teachers.
Anecdotal evidence that The Why Files is used With the support of NISE colleagues, The and enjoyed by many arrives daily. Some samples Why Files continued to draw together experts of readers’ comments: on the news media and science • “Serendipity or divine intervention. communication as well as the scientific Whatever the reason behind the find, I community to forge a product that transcends happened upon the Why Files. You've the interests of any single institution or done a fine job with varied subjects; the organization. The work of The Why Files articles are well done, full of wit, continued in that vein as the program made educational, and a pure delight (despite the the transition from NISE to the University of sometimes straight truth that may create Wisconsin–Madison Graduate School. We thoughts of becoming a vegetarian). anticipate that The Why Files will maintain its • “Way to go Why Files! Why do I enjoy global perspective and ties to NISE and NSF. the Why Files so much? Why me, Lord?” • “I am doing a science Web site review for The Research Component school and was really impressed with your Web site. Especially with the way you Over the four-year period that the CMA Team present both sides of evolution vs. has been conducting empirical research, we creationism. Anyway, you guys are getting have had a number of accomplishments. a great review from me!” Below we list our accomplishments in a rough • “Good site! It doesn't treat me like a chronological order, grouped by major moron.” research projects. • “I am a home educator and have just ‘discovered’ your page. It is great. Thank Review of Literature on Uses and Cognitive you.” Effects of Hypermedia The Why Files continues to build collaborations with other organizations. The team worked The first stages of our research program in closely with the NSF Office of Legislative and Year 2 began with an extensive review of the Public Affairs to produce a package in concert literature examining how individuals use the with the 1999 and 2000 National Science and technology of hypermedia—the foundation of Technology Weeks. Requests for links are the World Wide Web—to learn. Integrating routine, and numerous schools, businesses and research from fields as diverse as educational
53 psychology and technology, computer science, We also conducted a scientific survey of cognitive psychology, geography, library and repeat users of the site to better understand the information science, and communication, we demographic profile of The Why Files user. developed a review of the literature on how We found that, much like users of traditional individuals use hypermedia and its effects on their print science magazines, and much like the learning. This review, an early version of the users of the Web at the time (1997), repeat paper that won a top-three paper award from the users of The Why Files tended to be male, Communication Technology & Policy division of relatively young, well-educated, and very the Association for Education in Journalism & interested in science topics. It was unclear Mass Communication (AEJMC), included a from this initial finding whether or not these discussion of theories of information processing demographic characteristics were due to the that could inform research on learning about nature of access to the Web at the time, the science from Web sites like The Why Files. nature of individuals who tend to be interested Evidence pointed to a number of important in science, or some combination of the two. variables that might moderate or mediate the effects of this new technology, such as the design The results of these two studies were initially of Web sites and the expertise and goals of users. published by the NISE as an NISE Brief. A This review of the literature was published by the much more detailed analysis of these data was NISE as a Research Monograph. A revised later published in Public Understanding of versionisinpressatCommunication Yearbook, Science, an international journal specializing the major outlet in the field of communication for in informal science learning research. extensive literature reviews. Think-Aloud Analysis of Web Site Navigation Audit Trail Analysis of Web Site Navigation; and Information Processing Survey of Repeat Users of The Why Files Our finding of linear navigation patterns In Year 2 we conducted a number of studies to required a more detailed analysis of process evaluate The Why Files Web site. The first of data to better understand their implications. these studies unobtrusively tracked natural user Therefore, in Year 3 we designed and navigation throughout The Why Files site. During conducted a study in which we observed a three separate two-week periods, we developed a sample of adults using science information on data base that allowed us to observe patterns of the Web and thinking aloud as they did so. movement within and among science stories on This study initially allowed us to provide the site, as well as movements from outside of the advice to The Why Files creative staff about site into it, and from inside following links to how the site was being used and particular other Web sites. Contrary to the proposed design features that caused users problems. In advantages of hypermedia systems like the World effect, this study served as a usability analysis Wide Web, we found that the majority of users of The Why Files. Based on a summary of our did not make frequent use of nonlinear navigation findings, the creative staff of The Why Files features within the site or links to additional redesigned the home page of the site as well as information outside of the site. Instead, they a number of its other features. tended to follow linear navigation patterns, in many ways using The Why Files Web site like a The results of our think-aloud study also traditional print magazine. enabled us to analyze the information processing behaviors of individuals as they
54 navigated through Web sites. We developed a to Journal of Broadcasting & Electronic coding scheme to categorize thoughts as Media, the major outlet of the Broadcast indicating different types of information Education Association. processing behaviors and examined how these thoughts varied across a number of variables. The Experimental Analysis of Elaboration results of this research were published in Media Instructions on Learning from Web and Print Psychology, a journal devoted to research on the mass media from a psychological perspective. Another series of studies, begun late in Year 3 and continued through Year 5, examined the Experimental Analysis of Learning from Web vs. influence of various manipulations of the Print Presentations elaborative processing of individuals as they learned about science, primarily from Web Once we understood the demographics of users of sites but in some cases also print. Numerous science Web sites like The Why Files, and studies in education have indicated that understood their navigation behaviors through learning is increased substantially by these sites and their information processing elaborative processing of new information. behaviors during navigation, we expanded our However, these studies tend either to focus on research to understand how using the Web to very micro-level learning situations (e.g., communicate science might influence learning. paired associate learning; short prose Thus, during Year 3 and Year 4 we conducted paragraphs) or to use manipulations that are several studies of varying designs using various extremely powerful (e.g., semester-long stimulus materials to examine how science learning strategies training). We have learning might vary on the Web versus in print, endeavored to design elaboration holding content constant. These studies also manipulations that could be incorporated into manipulated variables such as user motivation science Web sites to increase elaboration (i.e., intentional vs. incidental learning) and Web among users. Our outcome measures have site design. Our findings suggest that learning varied from measures of immediate may suffer when information is communicated via recognition in some studies to measures of the Web compared to traditional print media. immediate and delayed cued recall in others. However, some Web site designs, particularly those that provide both nonlinear navigation Our results, which have not yet been fully options and instructional advisement, are not analyzed, suggest that there are a number of significantly worse for learning science complexities that may limit the effectiveness information than their traditional print of elaboration manipulations in real-world counterparts. Web sites. Early findings suggest, for instance, that Web sites themselves increase Incidentally, during the time of these studies The elaboration compared to the same content in Why Files used just this type of design. traditional print. This finding is encouraging and is consistent with some theories of Web We are currently submitting the findings of these effects. However, other findings suggest that various studies for presentation and publication. the elaboration manipulation may not be The first is currently under second review at effective on the Web because of a ceiling Communication Research, probably the most effect. Further, depending on the wording and selective journal in the area of mass placement of elaboration manipulations in a communication. Another has just been submitted text, learning of the elaborated passages may
55 increase at the expense of other information that site design on self-reported information did not receive the elaboration prompt. During the processing strategies, but also to determine next year we will further analyze these data in from observational data whether site design hopes that we can better understand the role of influences information processing. We will elaboration in learning from the Web. We will also be better able to identify more microlevel submit a manuscript or manuscripts to an features of site design that may increase educational technology or educational psychology beneficial forms (e.g., elaboration) or harmful journal. forms (e.g., disorientation) of information processing. We will also be able to Experimental Analysis of Web Site Structure empirically link information processing as Effects on Knowledge Structure measured using the think-aloud technique with measures of both content knowledge and Our most recent research (Year 5) has examined knowledge structure, something our previous how different Web site designs, specifically the work was unable to do. We expect that, use of linear versus nonlinear design structures, without further funding, it will take us can influence information processing, content approximately one year to analyze these data knowledge (as tapped by measures of recognition and produce conclusions. and of free and cued recall), and knowledge structure (interconnectedness and organization). III. Staff and Collaborators
Once these data have been coded and entered, we Terry Devitt, Sharon Dunwoody, and William hope to analyze them to determine whether the P. Eveland, Jr. (Team Leaders), Yael Gen, Sue design of Web sites can influence information Medaris, HeeSun Park, Jennifer Pearson, processing, as was suggested in our first think- Darrell Schulte, David Tenenbaum, Amy aloud study. In addition, since all our previous Toburen, Susan Trebach, and Tom Wiggins. research using measures of content knowledge have found either that print media are superior for Product learning, or at least that Web sites are not superior for learning, we have included detailed measures The Why Files Web site: of knowledge structure. These measures allow us http://whyfiles.news.wisc.edu/ to test the hypothesis that, while Web sites do not increase learning of content compared to print, different Web site designs can have effects on the NISE Publications of the Communicating structure of knowledge, potentially more with Mass Audiences Team important than effects on the content of knowledge. We expect that, without further Eveland, W. P., Jr., & Dunwoody, S. (1998). funding, it will take us approximately one year to Surfing the Web for science: Early data on the analyze these data and produce conclusions. users and uses of The Why Files (NISE Brief Vol. 2, No. 2). Madison: University of Think-Aloud Analysis of Web Site Structure Wisconsin–Madison, National Institute for Effects on Navigation and Information Processing Science Education.
We have also manipulated Web site structure and Eveland, W. P., Jr., & Dunwoody, S. (1999). user goals in a think-aloud study of adults. This Using hypermedia research to advance the study allows us not only to assess the impact of study of learning on the World Wide Web
56 (Research Monograph No. 15). Madison: Eveland, W. P., Jr., & Dunwoody, S. (2000, University of Wisconsin–Madison, National June). A test of competing hypotheses about Institute for Science Education. the impact of the World Wide Web versus traditional print media on learning. Paper NISE-Related Publications of the presented to the Information Systems division Communicating with Mass Audiences Team at the annual meeting of International Communication Association, Acapulco, Eveland, W. P., Jr., & Dunwoody, S. (1997, Mexico. August). Applying research on the uses and effects of hypermedia to the study of the World Eveland, W. P., Jr., & Dunwoody, S. (in Wide Web. Paper presented to the Communication press). Applying research on the uses and Technology & Policy division at the annual cognitive effects of hypermedia to the study of meeting of the AEJMC, Chicago. the World Wide Web. Communication Yearbook #25. Eveland, W. P., Jr., & Dunwoody, S. (1997, October). Communicating science to the public Eveland, W. P., Jr., & Dunwoody, S. (2000). via “The Why Files” World Wide Web site. Paper A test of competing hypotheses about the presented to the International Conference on the impact of the World Wide Web versus Public Understanding of Science and Technology, traditional print media on learning. Chicago. Manuscript submitted for publication.
Eveland, W. P., Jr., & Dunwoody, S. (1998). Eveland, W. P., Jr., & Dunwoody, S. (2000). Users and navigation patterns of a science World An investigation of the cognitive mediators Wide Web site for the public. Public and moderators of learning from the Web Understanding of Science, 7, 285-311. versus print. Manuscript submitted for publication. Eveland, W. P., Jr., & Dunwoody, S. (1999, August). Examining information processing on Millar, T., & Porter, A. (1998, January- the World Wide Web using think-aloud protocols. March). Harnessing the Web for educational Paper presented to the Communication impact: The science behind the news. Technology & Policy division at the annual Envision, 13(1), 20-21. meeting of the AEJMC, New Orleans, LA. Tenenbaum, D. (1997, Summer). Cyber scribe Eveland, W. P., Jr., & Dunwoody, S. (2000). tells all: Writing for the web. SEJournal Examining information processing on the World [Online]. Available: http://www.sej.org/ Wide Web using think-aloud protocols. Media sejournal/sej_su97.html Psychology, 2, 219-244.
57 58 Organizational Process Programs
Cognitive Studies of Interdisciplinary Collaboration
I. Team Mission NISE’s interdisciplinary teams in action. We conducted a two-year video ethnography of A major strategy of the NISE has been to use specified NISE teams and conferences in action, interdisciplinary teams to work on the collecting data suitable for later analyses by Institute’s mission of improving SMET researchers, both within and outside the NISE, education. Within the Institute, experts from a who are interested in developing explanations of variety of disciplines form working teams, of how groups construct knowledge and how social, various durations, that study significant issues physical, and cognitive constraints affect their and propose and carry out projects related to work. This report describes our progress on theory NISE goals. Such interdisciplinary teams are development, research, and data collection, which ubiquitous in educational research. They are comprised the bulk of our work during the three- increasingly common in industry, government, year funding period. The CSIC team did not and society in general. The National Science operate in years 4 and 5 of the NSF-supported Foundation encourages formation of such NISE. teams through their funding guidelines. For example, EHR’s ROLE program and the II. Accomplishments Interagency Education Research Initiative (IERI) encourage formation of Development of Foundational Theory interdisciplinary research teams. We grounded our work in a review of scientific Although a body of literature and much literature about how individuals and groups carry “wisdom of practice” have emerged in recent out cognitive activity and how that activity is years (e.g., Chubin, Porter, Rossini, & shaped by physical and social contexts. The Connolly, 1986; O’Donnell, DuRussel, & literature review synthesized research from social Derry, 1997), we still have little understanding psychology, cognitive psychology, small group of how social and cognitive processes interact research, and other fields, highlighting major to drive intellectual growth and construction issues influencing cognitive and social processes of products in natural interdisciplinary groups in interdisciplinary teams (O’Donnell, DuRussel, researching important social problems. Such & Derry, 1997). Important characteristics of knowledge might lead to better facilitation effective work groups were described, and origins technologies for improving interdisciplinary of difficulties within such groups were discussed. inquiry, which is known to be a difficult We determined, however, that much research on enterprise. group work has focused on groups that come together only in the laboratory, and that these The main goal of the Cognitive Studies of groups differ from true interdisciplinary groups in Interdisciplinary Communication (CSIC) team important ways. We concluded that study of was to better understand interdisciplinary interdisciplinary inquiry must focus more on collaboration in educational research. To this natural groups that experience special difficulties end, our team reviewed research, constructed because of institutional constraints, the theory, and conducted observational studies of combinations of disciplinary cultures represented,
59 and the implications of disciplinary allegiance the topic of interdisciplinarity. A number of to problem representations and solution questions guided discussion and activities during strategies. Our review examined several the conference. examples of interdisciplinary teamwork. Methodologies for studying interdisciplinary • What is the current theoretical base for interaction were suggested. To fully understanding interdisciplinary collaboration? understand and influence group processes in • What is already known about interdisciplinary interdisciplinary groups, we argued there is collaboration that can be applied to NISE need for a broad model of cognition within work and beyond? which to interpret the individual-level and • What are the best known methods for studying system-level variables and the interactions interdisciplinary collaboration? among these variables. • How can current and developing research findings contribute to the enhanced Later in the CSIC program such a model was performance of interdisciplinary teams? proposed, the Knowledge Building • What technological enhancements and tools Community (KBC) model, which is described might be used to facilitate teamwork within in two publications (Derry & DuRussel, 1999; the NISE? Derry, Gance, Gance, & Schlager, 2000) and • How might answers to the above questions is just beginning to be adopted by other help other research and educational reform research teams. partnership teams?
Conferences and Outreach Activities Following an opening address by Dr. Julie Thompson-Klein of Kent State University, To further understanding of interdisciplinary participants began by exploring what is meant by teamwork and to help refine research the term “interdisciplinarity” and identifying the objectives for this field of study, the CSIC key variables associated with interdisciplinary team hosted an invitational interdisciplinary collaboration. The keynote address by Dr. Gavriel conference, cosponsored a national cognitive Salomon of Haifa University, Israel, a noted science conference devoted to the theme of expert in the field, focused the conference on the interdisciplinarity, and participated in a cognitive processes of interdisciplinary teams and community outreach activity featuring current understanding of distributed cognition. members of the Sesame Street production Several case studies, some based on developing group. technologies that support teamwork, were presented. Participants then broke up into five Conference on Understanding working groups to explore, in more depth, key Interdisciplinary Teamwork. Thirty scholars questions derived from those noted above attended a small two-day conference in mid- concerning the role of theory, what the state of the November, 1996, entitled “Understanding science is, what the roadblocks are, and the most Interdisciplinary Teamwork: Challenges for pressing and important questions for researchers Research and Practice.” Conference in the study of interdisciplinary collaboration, as participants included faculty members of well as the potential for creating effective NISE, individuals who have special expertise environments for collaborative work through the or experience pertaining to interdisciplinary use of technology. As the conference closed, collaboration and problem solving, and participants acknowledged their own needs as a representatives of research projects addressing community of researchers studying
60 interdisciplinary collaboration to meet more team artifacts such as email, agendas, and visual frequently to share research findings and aids. This database documents the work and ideas. development of both short-lived and long-term interdisciplinary teams within the NISE. About 60 A conference report was prepared based on hours of tape have been transcribed and annotated the presentations and working group reports for specific analyses. In the future we will open and discussions at the conference. The this database to a limited number of outside conference was very positively evaluated by researchers, who will help conduct analyses of the LEAD Center. interdisciplinary teamwork.
Cognitive Science Society 1998 Annual To document our data collection effort, we Meeting. Interdisciplinary scholarship was the developed a technical manual that describes data theme of the Twentieth Annual Meeting of the collection, storage, transcription, and indexing: L. Cognitive Science Society, which took place A. DuRussel and L. L. Gance, 1997,. Technical at the University of Wisconsin-Madison, Manual for the Cognitive Studies of August 1-4, 1998. This conference, attended Interdisciplinary Collaboration Project. Included by 450 international scholars, was in the manual is our “master tape list” which cosponsored by NISE and cochaired by documents individual pieces of data and status Sharon Derry, the CSIC team leader. In information. The manual illustrates the addition to a conference proceedings organization of the database, what is available in (Gernsbacher & Derry, 1998), the meeting it, and how it is being indexed for access. produced a set of special thematic papers that were organized into a collection and will be Summary of Selected Research Findings published by Erlbaum (Derry & Gernsbacher, 2000). We conducted studies of two teams in the NISE— the Strategies for Evaluating Systemic Reform Sesame Street Unpaved. The CSIC team (SESR) team, and a working subcommittee of the sponsored a graduate symposium with College Level One (CL-1) team. In both studies members of the Sesame Street production we examined the genesis and development of group, which visited campus for an team knowledge over a lengthy period of time. anniversary performance. The symposium, We examined what roadblocks teams encountered “Understanding Collaborative Work: The and what constraints shaped their teamwork. Children’s Television Workshop as a Case Whole working groups were the major units of Study in Successful Interdisciplinary analysis. Our primary method was discourse Practice,” took place on September 22, 1998, analysis of meeting conversations. These analyses at the University of Wisconsin–Madison, and focused on collaborative conversations that could was advertised to faculty and students. be observed and recorded, tracing their impact on the papers, reports, and other products produced Data Collection by the teams. Our approach was largely qualitative in the sense that we descriptively We collected observational and interview data analyzed team processes to contribute to theory on three teams within the NISE. These data about effective collaborative work and to further include approximately 1,000 hours of video knowledge about how interdisciplinary and audio recordings of meetings, perspectives influence team functioning. We conferences, and interviews, as well as other focused less on the private cognitions of
61 individual members, although we did examine novice status as newcomers. Still, in order for these selectively through interviews. Detailed team members to work together, they must learn findings of our work are provided in the from one another and develop a common research reports and publications listed at the language. Although this process is not yet fully end of this section. Findings from selected understood, “apprenticeship” does not appear to studies are summarized briefly here. be an adequate metaphor for describing the learning that takes place among high-status team Sociocultural Theory and Interdisciplinary members from different disciplines. The need for Team Building. One study (DuRussel & a new theory became evident. Social scientists Derry, 1996) examined the validity of who study group interaction should be interested sociocultural theory (e.g., Wertsch, 1991) as a in this study, a critique of a currently popular viewpoint for analyzing data from the first few theory as a basis for research on interdisciplinary meetings of the SESR team. The major goal of process. the analysis was to determine whether the Team data “fit” categories and concepts A Case Study of an Interdisciplinary Team Trying proposed by sociocultural theory as lenses for to Understand Systemic Reform. This qualitative examining interaction during the team case study by Derry, DuRussel, and O’Donnell building process. We concluded that (1998) employed group information processing sociocultural theory did not adequately theory (e.g., Hinsz, Tindale, & Vollroth, 1997; describe major aspects of the team building Smith, 1994) as a lens for examining the SESR process we observed, leading us to search for team’s dynamics and processes, shedding light on a more appropriate theoretical viewpoint. how those dynamics and processes likely influenced the Team’s effectiveness as a We found that some knowledge-building knowledge building group. The primary processes predicted by sociocultural theory, contribution of this study is its detailed theory- namely those of apprenticeship and based description of how an interdisciplinary negotiation, did occur occasionally during team, composed of high-status academic advisors interdisciplinary collaboration. Wertsch’s constrained by limited time, interacted with a concept of “voices” was also useful in team leader and his staff to foster interdisciplinary revealing differences between social and knowledge construction. The analysis highlighted physical scientists and their strategies for difficulties inherent in trying to achieve a truly finding common ground. However, distributed form of information processing under sociocultural theory had only limited this form of team organization. Both limits and applicability to the kinds of interdisciplinary strengths of the Team’s knowledge construction teams found in the NISE. For example, processes were revealed. sociocultural theory implies that successful interdisciplinary teams operate like The Team’s difficulties were of three types. First, apprenticeship communities according to a the group had difficulty translating discussions process in which high-status old timers induct into products that reflected those discussions. lower-status newcomers. Status differences do Second, work accomplished at one meeting often in fact occur on interdisciplinary research did not serve as input to subsequent meetings; teams, but they probably are not associated hence the Team had difficulty building on and with learning the ropes in a team community. extending accomplishments. Third, both team and Many members of interdisciplinary teams are team leader sometimes rejected recommendations engaged for their expertise and do not have and agendas offered by the other.
62 However, the Team leader was a reflective that an inductive analytical approach was manager who was able to adjust team necessary—did not alleviate the conflict. organization and functioning as our analyses and other feedback became available. This study identified several factors contributing Moreover, significant knowledge building that to mental model misalignments. First, as combined ideas from different disciplines expected, individuals’ differing backgrounds contributed by team members did occur, (including their disciplinary history and primarily within the mind and work of the experience with tools such as software) influenced Team leader. The Team leader thus became the mental models they brought to the task. In the locus of successful interdisciplinary addition, the distribution of tasks among team knowledge construction because he valued the members caused each participant’s mental models Team and endeavored to synthesize input to be shaped by different goals, tools, and task selectively and thoughtfully. experiences. While these misalignments might have been resolved through group Analogical Reasoning and Mental Models in a communication, such communication did not Natural Working Group. A yearlong study, occur. The lack of articulation of the task conducted by Lori DuRussel as her master’s (including the ambiguous nature of the pathway thesis, produced several key findings. First, analogy) and the lack of known research team members held different mental models of approaches hampered team members’ attempts to the task distribution (who was responsible for work together successfully. Inconsistent doing what). Second, misalignments in attendance also may have prevented significant participants’ mental models of the task itself alignment from occurring. contributed to conflicts. Finally, it is crucial for interdisciplinary teams to bring team This analysis had several implications for members’ mental models of tasks and task management of interdisciplinary teams. Given the distribution into some degree of alignment, so diverse nature of interdisciplinary teams, some participants can work together successfully. degree of mental model misalignment is to be expected and possibly desired, since it The Team was attempting to analyze students’ theoretically enhances the wealth of perspectives transcript data to understand how students, that can be brought to bear on a problem. especially minorities and women, move However, it is important that the task be through college curricula in science and negotiated and/or articulated such that participants mathematics fields. The Team used the come to share an understanding of the crucial analogy of a “pathway” to characterize their steps involved and the expected approach that is viewpoint, and this viewpoint shaped thinking to be taken. In addition, task distribution should about their data analysis. Ambiguities in be clarified so that all team members are aware of participants’ definitions of what pathways who is responsible for what parts of the project. were and how they should be investigated Such articulation and clarification requires resulted in conflict. In addition, because attention to communication. One key to ensuring participants’ models of software’s role in the adequate communication is as simple as insisting task were not well aligned, the analyst used on consistent attendance at team meetings, thus existing software in a way that did not match ensuring that all team members are influenced by other participants’ expectations, and his work the same interactions and increasing the chances was criticized. Even apparent alignments in that participants’ models will evolve in mental models—as in participants’ agreement compatible ways.
63 References Derry, S. J. (October, 1997). Cognitive case studies of interdisciplinary teamwork. Seminar Chubin, D. E., Porter, A. L., & Rossini, F. A. presented for the Department of Educational (1986). Interdisciplinary research: The why Psychology at the University of Wisconsin– and the how. In D. E. Chubin, A. L. Porter, F. Madison. A. Rossini, & T. Connolly (Eds.), Interdisciplinary analysis and research: Derry, S. J. (1997). Brownbag session presented Theory and practice of problem-focused at the National Institute for Science Education. research and development.Mt.Airy,MD: Lomond. Derry, S. J. (1999). Toward a Knowledge Web for collaborative learning in secondary Hinsz, V. B., Tindale, R. S., & Vollrath, D. A. teacher education (NISE Brownbag Lecture). (1997). The emerging conceptualization of Madison, WI: University of Wisconsin-Madison, groups as information processors. National Institute for Science Education. Psychological Bulletin, 121, 43-64. Derry, S. J., & O’Donnell, A.M. (1996). Smith, J. B. (1994). Collective intelligence in Distributed and individual cognitions in computer-based collaboration. Hillsdale, NJ: interdisciplinary collaboration. Paper presented at Erlbaum. the annual meeting of the American Educational Research Association. Wertsch, J. V. (1991). Voices of the mind: A sociocultural approach to mediated action. Derry, S. J., Gance, S., & Gance, L. L. (1998). Cambridge, MA: Harvard University Press. Cognitive conflict in interdisciplinary teamwork: A case study. Presentation at SRI International. III. Staff and Collaborators DuRussel, L. A. (1998). Analogical reasoning in an interdisciplinary working group. Unpublished Staff: Sharon Derry (Team Leader), Lori master’s thesis, University of Wisconsin– DuRussel, and Laura Lee Gance. Madison, Madison, WI. Fellow: Angela O’Donnell DuRussel, L. A., & Derry, S. J. (1998). Mental models in an educational research team. Poster Collaborators: Jan O’Neill and Mark Schlager. presented at the annual meeting of the American Educational Research Association, San Diego, CA. NISE Publications and Presentations of the Cognitive Studies of Interdisciplinary Gance, L. L. (1998). Understanding Collaboration Team interdisciplinary teamwork: Challenges for research and practice: The report of a conference Derry, S. J. (August, 1997). Knowledge organized by the Cognitive Studies of construction in interdisciplinary teams: Interdisciplinary Collaboration Project, Madison, Strategies and roadblocks. Invited address at WI, 15-16 November 1996 (Workshop Report No. the annual meeting of the American 5). Madison: University of Wisconsin–Madison, Psychological Association, Chicago, IL. National Institute for Science Education.
64 O’Donnell, A. M., DuRussel, L. A., & Derry, Derry, S. J., & Gernsbacher, M. (Eds.). (2000). S. J. (1997). Cognitive processes in The problems and promises of interdisciplinary interdisciplinary groups: Problems and scholarship: Perspectives from cognitive science. possibilities (Research Monograph No. 5). (University of Wisconsin–Madison). Manuscript Madison: University of Wisconsin-Madison, in preparation. National Institute for Science Education. DuRussel, L. A., & Derry, S. J. (1996). NISE-Related Publications of the Cognitive Sociocultural approaches to analyzing cognitive Studies of Interdisciplinary Collaboration development in interdisciplinary teams. In Team Proceedings of the Eighteenth Annual Meeting of the Cognitive Science Society (pp. 529-533). Derry, S. J., DuRussel, L. A., & O’Donnell, A. Mahwah, NJ: Erlbaum. M. (1998). Individual and distributed cognitions in interdisciplinary teamwork: A DuRussel, L. A., & Derry, S. J. (1998). developing case study and emerging theory. Analogical reasoning in a natural working group. Educational Psychology Review, 10(1), 25-56. Proceedings of the Twentieth Annual Meeting of the Cognitive Science Society. Derry, S. J., Gance, S. P., Gance, L. L., & Schlager, M. (2000). Toward assessment of DuRussel, L. A., & Derry, S. J. (2000). Mental knowledge building practices in technology- models of teamwork. (University of Wisconsin– mediated work group interactions. To appear Madison). Manuscript in preparation. in S. Lajoie (Ed.), Computers as cognitive tools II: No more walls. Mahwah, NJ: Gernsbacher, M. A., & Derry, S. J. (Eds.). (1998). Erlbaum. Proceedings of the twentieth annual meeting of the Cognitive Science Society. Mahwah, NJ: Erlbaum.
65 Information Resource Coordination
I. Team Mission dissemination to the public. By early 1999, the online Web presence of the NISE was well The mission of the Information Resource established, but was difficult for users to recall or Coordination (IRC) Team has been to find through commonly used search engines. In examine issues in information technology use addition, the large stock of NISE publications and knowledge sharing (including information were not available for online viewing. The technology systems design) to support activities initiated or supported by the IRC Team improved information flow for science and include the following: mathematics education. There have been two focus areas of the mission: information flow Registration of simpler Web address. The formal and coordination of NISE projects themselves, Web site URL (universal resource locator) for the and the application of research knowledge and NISE is http://www.wcer.wisc.edu/nise/.After design engineering to improve information discussions with campus representatives, it was sharing in education communities of practice. determined that there was no conflict of interest to register a simpler domain name for the benefit of The IRC Team was formed in spring 1999 by visitors to the site. The name “nise.org” was Associate Director Barrett Caldwell, based on available and was registered as an alternative path his research and teaching interests in to the NISE, http://www.nise.org. This address information and communication technology takes visitors to the same NISE home page as the systems engineering. As a result, the IRC Team longer address above. has played a multifaceted role, including consulting to other NISE teams (Secondary Appropriateness of online dissemination. Some Teacher Education Project, Systemic Reform academic disciplines and professional Project), front-office coordination and organizations have resisted online publication or implementation projects, and direct action dissemination of scholarly works due to concerns research with educational practitioners. about intellectual property and copyright infringement. These concerns had prevented NISE The IRC Team accomplishments will be listed from making many of its publications available in three sections: NISE Information online. A new evaluation of the practice of online Coordination Activity, NISE Project Support publication was initiated by IRC Team, and, as a Efforts, and Independent Research and result, more NISE Briefs, Reports, and Development Tasks. Because of the Proceedings are available in abstract or complete distribution of unique effort by IRC Team form from the Publications section of the Website. members, the third accomplishment area will be the longest. NISE Project Support Efforts
II. Accomplishments During the 1999-2000 period, IRC Team members provided critical information, expertise, NISE Information Coordination Activity and research capability resources to two other NISE Teams: the Secondary Teacher Education The primary coordination activity in this Project (STEP: Sharon Derry, Team Leader), and accomplishment area has been focused at the Systemic Reform / Milwaukee Schools Team improving NISE information availability and (InfoNet: Norman Webb, Team Leader). Because
66 IRCT members were functionally integrated STEP required substantial detailed development into these other teams, relevant project activity to enable its effective use by educational will also be summarized in those teams’ psychology students in their ongoing coursework. reports. This development work was begun by both Nicole Canty and Jeff Watson and was continued through STEP Support.STEPhasbeenfocusedonthe June 2000 by Nicole Canty. development of a “knowledge Web” of resources for preservice and inservice teacher InfoNet Support. The Systemic Reform/ development in the learning sciences. The Milwaukee Schools InfoNet project led by results of the knowledge Web include a novel Norman Webb has identified a critical Web site design to provide case-based information-sharing problem at the level of instructional support for cognitive science and individual schools’ access, use, and decision- learning theory topics in education courses. making capability of data and resulting The human-computer interaction (HCI) information produced by the schools themselves. design, implementation, and evaluation of that This disconnect between the ability to provide site was conducted in large part by IRC Team data annually or semi-annually to the larger member Nicole Canty. A conceptual model of school district, and the ability to effectively utilize the overall information resource development local information at shorter time intervals, was a project is shown in Figure 2. primary focus of the support provided by IRC Team members Jeff Watson and Susan Zeyher.
Teacher Professional Development Resources
Knowledge Web Software Tool Box Domain and Project/ Foundations Course Resources and Theories
Case Learning Modules Examples and Student Projects
NISE Site, with Links to Content, Classrooms, and Collaborators
Figure 2. Conceptual model of Teacher Professional Development Information Resources used by IRC Team
67 Descriptions of the design of an effective The SMELT project did not simply result in a set InfoNet will be found in more detail in the of tools for practicing teachers. Both the TAPPED Systemic Reform Team’s reports. However, IN Outreach and SMELT CD Development tasks critical elements of an effective InfoNet provided opportunities for undergraduate architecture include the availability of data at engineering students to participate in SMELT as the right time and grain size for local users. part of their educational experience. Members of Systems to provide appropriate data to schools the Society of Women Engineers (SWE) student or classrooms are not well integrated with chapter have participated as part of their chapter large scale educational statistics repositories outreach activities; students in HCI design and for policymaking or funding evaluations. This engineering senior design students have separation of local from general information participated in SMELT design tasks in formal has placed additional burdens on schools to classes as part of their design education. modify or invent their own data reporting and use systems. The IRC Team has provided TAPPED IN Outreach. The IRCT interactions with ongoing assistance to the InfoNet project to TAPPED IN began as part of STEP project describe the necessary and desirable interactions in late 1998. Since then, IRC Team characteristics of data systems and members have participated in TAPPED IN information flow needs for local schools; Outreach activities including the TAPPED IN InfoNet reports on these topics are in Carnival, held annually in July. In spring 1999, production. IRCT members Nicole Canty and Susan Zeyher participated in the design of a “virtual suite” for Independent Research and Development Tasks TAPPED IN librarian participants from Kentucky.
The primary demonstrable outcomes of the At the July, 1999, Carnival, IRCT members hosted IRCT are associated with the independent a “Web site swap” for teachers who were project task known as the Science, interested in sharing or learning about useful Web Mathematics, and Engineering Learning sites for various subjects. Over 75 sites have been Technologies (SMELT) project. SMELT has collected from TAPPED IN participants since then. involved direct outreach and cooperation with This list has been pared to approximately 45 sites teacher practictioners in the Teacher primarily focused on science and mathematics Professional Development Institute (TAPPED topics for middle and high school students. These IN), an online “education community of sites have served as the prototype content for the practice” hosted by SRI International. In SMELT CD Development task described below. addition, SMELT has produced a CD of teacher-relevant and “teacher-tested” (i.e., In spring 2000, the UW student chapter of the submitted by practicing teachers as useful and Society of Women Engineers (SWE) began a series high quality) Web sites with an improved of outreach activities with TAPPED IN search functionality based on an XML participants. These activities include (during the Metadata architecture (see below). The XML academic year) formal “office hours” and Metadata stands as a unique development task; scheduled question and answer periods for “Ask an however, it is integrated into the CD as a Engineer” to assist teachers and high school method for improving search and selection of students contemplating majoring in science and resources by teachers with slow or limited mathematics disciplines. High school to college internet access. transitions, women in science career role models, and general SWE outreach topics are among those
68 that have been discussed (see, for example, The SMELT CD, therefore, was designed for http://www.tappedin.sri.com/cgi- educators and others interested in quickly and bin/calendar/calendar.cgi?file=calendar&mon effortlessly finding information on the World th=April). This outreach activity has become Wide Web about science and mathematics integrated into SWE ongoing outreach plans education resources. From the users’ manual: and is now independent from IRCT involvement. The CD’s main resource is the “Accurate Search.” By clicking on this button, you can search a SMELT CD Development. The SMELT database included in the CD for a number of development concept of a CD to improve resources available on the World Wide Web. You teachers’ access and use of online information do not need to be connected to the Internet to search and evaluate the resources that are resources began during online discussions discussed in the CD database. You should connect with teachers in TAPPED IN. Their to the Internet only when you have made your complaints were that there were rarely choice of resource(s) and want to download them. teacher-oriented descriptions of useful In this way, you can be in any location and look materials that could be used in the classroom, for resources without connecting to the Internet and such materials were difficult to search for and dealing with long search times and a using commercial search engines. This confusing maze of information. problem is compounded for teachers from small and remote schools, who may have The SMELT CD design process included major limited time, bandwidth, or restricted access development steps in two Industrial Engineering (due to firewalls or other local site design courses. During the fall 1999 semester, two restrictions) for online searching for Internet project teams in IE 552 (Human Factors resources. Engineering Design and Evaluation) were responsible for designing the interface, and One of the strongest and most frequent support infrastructure, for a self-sustaining requests by members of the TAPPED IN SMELT resource. These project teams created the community has been for a central repository of initial interface structure for the SMELT CD, information, including Web sites that are including installation tools and user guides. suitable for classroom presentations and student use. Most teachers do not have the This first “alpha” prototype was field-tested with time, expertise, or awareness of the resources teachers from Madison, as well as during a site available on the Web to effectively search for visit to Roaring Fork High School, Carbondale, those resources among the tens of millions of CO. Results from this initial site visit were Web sites. A repository that can be trusted, brought back to IRC Team members to provide easily found, and easily used to bring teachers practitioner feedback for SMELT improvement. quickly to sites they can use is a critical Additional feedback was obtained during a spring technical priority for improving teachers’ use 2000 course project in the IE 692 Special Topics of the Web. Such a repository must in Human Factors design course. Usability data on incorporate a human technical support staff existing SMELT interface designs were collected able to assist teachers in its use, answer from several users, and the results were combined questions that are raised, and help to maintain with previous usability findings from Roaring the repository as a dynamic learning Fork High School and TAPPED IN designers to environment. produce the final SMELT CD prototype.
69 Educational XML Metadata. The major source efforts of Susan Zeyher) a productive and of improvements in information search and contributing member of The Dublin Core resource selection in the SMELT CD comes Education Working Group, providing insights, from a novel information architecture based suggestions, and contributions at a level on the emerging Extensible Markup Language commensurate with the national groups. As a (XML) standards. Developing an appropriate result, the IRCT XML Metadata can be architecture for use by teachers requires a considered a “world-class” architecture with focused development of “metadata” (data that capability for increasing development and describe, classify, and permit search), based application both at more detailed grains in specific on the needs and environment of projected science and mathematics disciplines, as well as users. The IRCT focus, unlike most XML parallel extensions to the social and behavioral standards, was on direct use by teacher sciences and humanities. practitioners for in-class resource use, rather than on educational administration or research Discussions are currently underway to license the applications. In addition, the goal of the IRCT XML, information architecture, and resource activity was to provide teachers with easily coordination functions developed in the SMELT identified and selected terms for searching project to a Wisconsin company providing science and mathematics resources. This direct classrooms with access to live reports and focus on teacher practitioners as end-users, as continuing availability of mission data from well as hierarchical search architectures for NASA missions to the moon, asteroids, and Mars. science and mathematics, sets the IRCT apart from most XML development efforts. III. Staff and Collaborators
The IRCT members used as references the Staff: Barrett Caldwell (Team Leader), Armand Dublin Core Education Working Group and Ardika, Nicole Canty, Virginia “Katie” Emery, the national XML standards being developed Christine Lee, Nualwan Narong, Enlie Wang, Jeff in the U.S. (GEM), Australia (EdNA), and the Watson, and Austen Zuege. European Union (ARIADNE). By spring 2000, the IRCT had become (through the Fellow: Susan Zeyher
70 Formative Evaluation
I. Team Mission conclude the investment has more than paid for itself. A research center of NISE's size and The objectives of the NISE Formative challenging scope of work is difficult to create Evaluation team evolved over the course of and maintain. Time is of utmost importance; the NISE’s history. The first objective was to productivity must begin immediately and increase provide the NISE leadership with real-time over time. NISE's commitment to interdisciplinary information about the development of the work across the boundaries of science, Institute as an organization to guide midcourse mathematics, social science, and education added corrections and thus maximize its goal to the challenge. The formative evaluation team achievement. By the middle of Year Three, efforts identified key problems as they arose and the NISE team leaders agreed that the NISE pointed the way to timely solutions. had evolved to the point where the value of intensive formative evaluation reached the II. Accomplishments point of diminishing returns. The formative evaluation work was sustained at a low level Formative Feedback on the NISE as an in Years Four and Five. During Year Five the Organization formative evaluation objective was completed with the production of The Role of Formative The feedback that the Formative Evaluation Team Evaluation in the Development of an provided during the first three years helped Interdisciplinary Academic Center (NISE Institute leaders understand how various NISE Occasional Paper No. 8). A second objective constituents were experiencing and viewing the of the Team was to provide formative and NISE teams and the NISE overall (NISE Internal summative evaluation of the NISE Forums. Document Nos. 1, 3, 4, 5, and 6). The The Team pursued this objective by evaluating Management Team, Team Leaders, and most team the First, Second, and Third Annual Forums members used these reports to make and the Graduate Education Forum. The organizational improvements. For example, the prototype for Forum evaluation having been Formative Evaluation Team’s feedback helped the established, the NISE Central Office assumed NISE leaders responsibility for this activity as of Year Four. • redefine their goal for including During Year Two, the Team began pursuing a individuals from heterogeneous third objective, which was to draw on its backgrounds, social science expertise to produce syntheses • redistribute their budget to fund people for and proceedings of the Second and Third a higher percentage of their time, and Annual Forums, and the Graduate Education • improve the quality of cross-team Forum. During Year Five, the Team pursued a communication processes by instituting fourth objective—an impact study of the retreats and a new members’ orientation College Level One Team’s Field-tested packet. Learning Assessment Guide (FLAG). The value of this formative evaluation process is NISE's investment in formative evaluation is described in “The Role of Formative Evaluation unique among education research centers and in the Development of an Interdisciplinary institutes. After 5 years of what we believe to Academic Center.” This paper, written during be an especially productive institute, we Year Five, explores the processes and emerging
71 principles by which one complex, cross- based on participant think pieces (short papers disciplinary organization—the NISE— prepared by forum participants while on site). The developed. In addition, it locates the synthesis comprises a key component of the development of the NISE in the context of the proceedings and was very positively reviewed. research literature on other cross-disciplinary During fall 1997, the Team joined the College organizations and makes a case for the utility Level One Team’s forum planning group. of formative evaluation for the development of Because the members of the Formative Evaluation academic centers. The primary purpose of this Team are researchers at the UW-Madison paper is to serve the NISE, NSF leaders and Learning through Evaluation, Adaptation and program officers, and others seeking to foster Dissemination (LEAD) Center, they brought productive multidisciplinary academic expertise developed through their SMET research centers. We are moving with ever evaluation research work. From February 1998 increasing momentum into a period when key through the beginning of Year Four (1998-99), the problems in science, mathematics, and Team produced the Year Three Forum engineering education can no longer be solved Proceedings, working closely with a CL-1 Fellow, by people from a single field of expertise, and Elaine Seymour. In addition, throughout Year few experts are trained to work in effective Three, the Formative Evaluation team participated cross-disciplinary teams. on the Graduate Education Forum planning group. During Year 4, the Team worked with the leader Forum Evaluation of the Graduate Education Forum to produce the Proceedings of this event. The Formative Evaluation team evaluated the first three annual Forums and the Graduate Formative and Summative Evaluation of the NISE Education Forum. These findings appear as College Level One (CL-1) Team’s Field-tested ReportNos.2,7,8,and9oftheNISEInternal Learning Assessment Guide (FLAG) Web Site Document Series. These reports enabled the organizers of each successive Forum to design The Formative Evaluation team conducted an national events that have improved over time impact study of one of the key products of the and have established a reputation for College Level One Team’s Field-tested Learning excellence with an increasingly wide national Assessment Guide (FLAG) Web site. The FLAG audience. is an online faculty development resource that provides visitors with an introduction to Forum Planning and Proceedings assessment, a description of a dozen or so alternative classroom assessment techniques Starting in the second year, the Formative developed by experts, and various examples of Evaluation team began supporting the NISE these techniques that can be downloaded and used by helping with the planning and analysis of (with some revision) in a course. The evaluation the annual Forums. During 1996-97, the Team was designed to determine how well the FLAG worked closely with the Systemic Reform and fits the needs of instructors who teach Interacting with Professional Audiences introductory science, math, engineering and Teams to plan and organize the Second technology (SMET) courses and to explore ways Annual Forum. In addition, the Formative in which instructors report using the assessment Evaluation team took major responsibility for methods in their courses as well as any new managing the production of the forum understanding of assessment that they develop as proceedings and produced a forum synthesis a result of using the FLAG. Evaluation data were
72 gathered through individual interviews with a No. 3 Formative Feedback Report 1.3, Baseline representative sample of 50 “reform ready” Report on NISE "Intermediaries," April 1996. faculty (the audience for which the FLAG is No. 4 Formative Feedback Report 1.4, Report on designed) who teach biology, chemistry, Non-Team Leader Team Management Team engineering, mathematics, and physics at Participants, April 1996. diverse types of institutions of higher No. 5 Formative Feedback Report 2.1 Report on education. The Team wrote a detailed Perspectives of the Co-Directors, Team formative feedback report for the College Leaders Team, and Management Team, Level One Team (see below) and will soon September 1996. submit a manuscript for publication as a paper No. 6 Formative Feedback Report 2.2, Report on intended for a broader audience. These Perspectives of the NISE Membership, April documents address technical aspects of 1997. participants' use of the FLAG, overall No. 7 Second Annual NISE Forum Evaluation reactions to current and future value of the Report, March 1997. FLAG, experience and reactions to the No. 8 Third Annual NISE Forum Evaluation individual sections of the FLAG, and Report, April 1998. suggestions for improving the FLAG. The No. 9 Graduate Education Forum Evaluation LEAD Evaluators also met with the College Report, August 7, 1998. Level One Team web master with a list of specific suggestions that came from the NISE Publications of the Formative Evaluation interview data for improving the FLAG Team design. Many of these changes have been made and have been influential in the creation Clune, W. H., Millar, S. B., Raizen, S. A., Webb, of the next College Level One product, the N. L., Bowcock, D. C., Britton, E. D., Gunter, R. Learning Through Technology website. L., & Mesquita, R. (1997). Research on systemic reform: What have we learned? What do we need III. Staff and Collaborators to know? Synthesis of the Second Annual NISE Forum, Volume 1: Analysis; Volume 2: Staff: Susan B. Millar (Team Leader), Proceedings (Workshop Report No. 4). Madison: Dianne C. Bowcock, Anne C. Burda, Sarah K. University of Wisconsin–Madison, National A. Pfatteicher, Ramona L. Gunter, Sarah Institute for Science Education. Mason, and Susan Daffinrud. Daffinrud, S. M., & Herrera, O. L. (2000). Products Formative feedback report: Faculty participants' experience, reactions to, and use of the Field- NISE Internal Documents tested Learning Assessment Guide (FLAG) Web site. Madison: University of Wisconsin–Madison, No. 1 Formative Feedback Report 1.1, National Institute for Science Education. Baseline Report on the Team Leaders Team and Management Team, October Millar, S. B. (Ed.). (1998). Indicators of success 1995. in postsecondary SMET education: Shapes of the No. 2 Formative Feedback Report 1.2 First future.Synthesis and proceedings of the Third Annual NISE Forum Evaluation Report, Annual NISE Forum (Workshop Report No. 6). April 1996. Madison: University of Wisconsin–Madison, National Institute for Science Education.
73 Millar, S. B. (2000). The role of formative Millar, T. S., Mason, S. A., Gunter, R. L., & evaluation in the development of an Millar, S. B. (1999). Synthesis of the science, interdisciplinary academic center (Occasional mathematics, engineering and technology Paper No. 8). Madison: University of Graduate Education Forum (Workshop Report Wisconsin–Madison, National Institute for No. 7). Madison: University of Wisconsin– Science Education. Madison, National Institute for Science Education.
74 Summary
The Institute’s goals are ambitious. We attack interconnections. A strong commitment to head-on three of the most important areas for dissemination insures that our work comes to improving SMET education. At the college the attention of the NSF, professional level, we are helping to define an emerging field organizations, practitioners, and other of SMET education R&D; at present, most researchers. education work in higher education is focused on innovation rather than research and is At the end of five years, we have launched a scattered. In teacher education, both preservice whole new approach for the continuous and inservice professional development, we improvement of SMET education. The goal of seek to reconceptualize the field, identifying high levels of SMET literacy for all segments of new and more effective practices and how they our population is better understood and more can be implemented successfully across the broadly accepted. New communities of country. In systemic reform, we seek to set the scholarship and practice have been established, research agenda so that this reform will be where scientists, education researchers, and pursued with increasing effectiveness through education practitioners work collaboratively to continuous improvement for decades to come. address the enduring problems of SMET Work in each of these areas enhances work in education, problems that have resisted solutions the others, as we focus on their from more narrow approaches.
75
Appendix A NISE Staff and Collaborators by Team
Systemic Reform Michael Knapp, Associate Professor, College of Education, University of Washington, Seattle Christopher Anderson, Professor of Astronomy, UW–Madison Sarah Mason, Associate Researcher, WCER, UW–Madison Steven F. Bauman, Professor of Mathematics, UW–Madison Robert Meyer, Senior Scientist, WCER, UW– Madison and Lecturer and Research Associate at Vicki M. Bier, Associate Professor of the Harris Graduate School of Public Policy Industrial Engineering and Nuclear Studies, University of Chicago Engineering, UW–Madison Susan Millar, Director, LEAD Center, UW– Jeffery Braden, Professor of Educational Madison Psychology, UW–Madison Jennifer O’Day, Professor, Department of Thomas Carpenter, Professor of Curriculum Educational Policy Studies, UW–Madison and Instruction (Mathematics Education), UW–Madison Eric Osthoff, Research Assistant, UW–Madison
Donald Chambers, Researcher, WCER, UW– Andrew Porter, Professor of Educational Madison Psychology and Director, NISE, UW–Madison
William H. Clune, William Voss-Bascom Senta Raizen, Director, National Center for Professor Emeritus of Law, UW–Madison Improving Science Education, Washington, DC
Janice Downer, Assistant Professor of Alberto J. Rodriguez, Assistant Professor of Engineering Mechanics and Astronautics, Curriculum and Instruction (Science Education), UW–Madison New Mexico State University, Las Cruces
Deborah Tepper Haimo, Department of Judy Roitman, Professor of Mathematics, Mathematics, University of California, San University of Kansas, Lawrence Diego Thomas A. Romberg, Sears Roebuck Foundation- Ron Jetty, Project Assistant, Education, UW– Bascom Professor in Education (Mathematics Madison Education), UW–Madison
Michael Kirst, Professor of Education and Pat Rossman, Teacher, Elvehjem Elementary Business Administration, Stanford University, School, McFarland, WI Stanford, CA Richard Rossmiller, Professor Emeritus of Educational Administration, UW–Madison
77 Bassam Shakhashiri, Professor of Chemistry, Mark Driscoll, Project Director, Education UW–Madison Development Center, Incorporated, Newton, MA
William Tate, Associate Professor of Kathy Dunne, Director of Professional Curriculum and Instruction (Mathematics Development, WestEd, Stoneham, MA Education), UW–Madison Joan Ferrini-Mundy, Professor of Mathematics Christopher A. Thorn, Associate Researcher, Education, University of New Hampshire and WCER, UW–Madison National Academy of Sciences, Washington, DC
Norman Webb, Senior Research Scientist, David Hartney, Managing Director, First Hand WCER, UW–Madison Learning, Incorporated, Buffalo, NY
PaulaA.White,ProjectManager,NISE,UW– Peter Hewson, Professor of Curriculum and Madison Instruction, UW–Madison
John Witte, Professor of Political Science, Gregg Humphrey, Project Director, WestEd, UW–Madison Willliston, VT
John Wright, Professor of Chemistry, UW– Susan Loucks-Horsley, Senior Research Madison Associate, National Center for Improving Science Education, Tucson, AZ (deceased) Professional Development Nancy Love, Dissemination Specialist, National Josefina Arce, Professor of Chemistry, Center for Improving Science Education, University of Puerto Rico, San Juan Washington, DC
Carne Barnett, Regional Director, National Mary Kennedy, Professor, Michigan State Council of Supervisors of Mathematics, University, East Lansing WestEd, Oakland, CA Barbara Miller, Project Director, Educational Virginia Bastable, Director, Summer Math for Development Center, Incorporated, Newton, MA Teachers, Mount Holyoke College, South Hadley, MA Jean Moon, Advisor to the Exxon Education Foundation, Exxon Corporation, Groton, MA Cathy Carroll, Regional Director, Math Renaissance K-12, WestEd, San Mateo, CA Susan Mundry, National Center for Improving Science Education, Andover, MA Karen Cerwin, Regional Director, California Science Implementation Network, Rim Forest, Jan Phlegar, Director, Learning Innovation at CA WestEd, Stoneham, MA
Kathy DiRanna, Mentor Coordinator, Lynn Rankin, Director, The Exploratorium, San California Science Alliance, WestEd, Irvine, Francisco, CA CA
78 Ann Rosebery, Principal Scientist, TERC, Gina Brissenden, CL-1 Team Member, UW– Cambridge, MA Madison
Susan Jo Russell, Principal Scientist, TERC, Aaron Brower, Professor of Social Work, UW– Cambridge, MA Madison
Mark St. John, President, Inverness Research AnnB.Burgess,Director,BiologyCore Associates, Inverness, CA Curriculum, UW–Madison
Deborah Schifter, Senior Scientist, James Cooper, Professor of Education, California Educational Development Center, Newton, State University, Dominguez Hills MA Patricia Cross, Professor of Higher Education, Vernon Sells, Mathematician, Ventures in Emeritus, University of California, Berkeley Education, Hespeus, CO Sue Daffinrud, Research Specialist, LEAD Jerome Shaw, Senior Research Associate, Center, UW–Madison WestEd,MenloPark,CA Samuel Donovan, Project Assistant, UW– Rita Starnes, Regional Director, California Madison Science Alliance, WestEd, San Mateo, CA Janice Downer, Assistant Professor of Mary Stenson, Administrative Assistant, Engineering Mechanics and Astronautics, UW– Learning Innovations, WestEd, Boston, MA Madison
Katherine Stiles, Research Associate, National Douglas Duncan, Associate Professor of Center for Improving Science Education, Astronomy and Astrophysics, University of Washington, DC Chicago, and National Education Coordinator of the American Astronomical Society Jo Topps, Regional Director, California Science Implementation Network, WestEd, Arthur B. Ellis, Professor of Chemistry, UW– Long Beach, CA Madison
Iris Weiss, Evaluator and President, Horizon Roscoe Giles, Professor of Engineering, Boston Research Incorporated, Chapel Hill, NC University, Boston, MA
College Level One Jack Husted, Project Assistant, Social Work, UW–Madison Clifford Adelman, Senior Research Analyst, U.S. Department of Education/OERI, Anthony Jacob, Associate Director of Chemistry Washington, DC Learning Center, UW–Madison
Briana Bright, Research Assistant, UW– Mary Kennedy, Professor of Teacher Education, Madison Michigan State University, East Lansing
79 Louise Liao, Program Director, Coalition for Mary Beth Stanne, Research Assistant and Education in the Life Sciences, UW–Madison Consultant, University of Minnesota, Minneapolis
Kate Loftus-Fahl, Science Writer, UW– Secondary Teacher Education Project Madison Brian Bietzel, Doctoral Student, Educational Robert Mathieu, Professor of Astronomy, Psychology, UW–Madison UW–Madison Barrett S. Caldwell, Associate Professor of Catherine Middlecamp, Faculty Associate, Industrial Engineering, UW–Madison Chemistry, UW–Madison Nicole Canty, Program Assistant, School of Susan Millar, Director, LEAD Center, UW– Engineering, UW–Madison Madison Sharon J. Derry, Professor of Educational Terrence S. Millar, Professor of Mathematics Psychology, UW–Madison and Associate Dean, Graduate School, UW– Madison Kate Hewson, Graduate Student, Educational Psychology, UW–Madison John W. Moore, Professor of Chemistry, UW– Madison Jong-Baeg Kim, Doctoral Student, Educational Psychology, UW–Madison Greg Moses, Associate Dean, Engineering, UW–Madison Julia Lee, Graduate Student, Educational Psychology, UW–Madison Jeanne Narum, Director, Project Kaleidoscope and The Independent Colleges Office, Paul Feltovich, Interim Chair, Medical Education, Washington, DC Southern Illinois University School of Medicine, Springfield, IL Pamela Robinson, Researcher, School of Education, California State University, Cindy Hmelo, Assistant Professor, Program in Dominguez Hills Learning Cognition and Development, Educational Psychology, Rutgers University, New Gloria Rogers, Vice President, Institutional Brunswick, NJ Research and Assessment , Rose-Hulman Institute of Technology, Terre Haute, IN Meg Meyer, Coordinator, Secondary Mathematics Teacher Education, UW–Madison Walter Secada, Professor of Curriculum and Instruction (Mathematics Education), UW– Susan Millar, Director, LEAD Center, UW– Madison Madison
Karl Smith, Associate Professor of Civil Roy Pea, Director, Center for Technology in Engineering, University of Minnesota, Learning, SRI International, Menlo Park, CA Minneapolis
80 Julie Posselt, Project Assistant, Curriculum Sharon L. Dunwoody, Director and Professor of and Instruction, UW–Madison Journalism & Mass Communication, UW– Madison Mark Schlager, Director, Teacher Professional Development, SRI International, Menlo Park, William Eveland, Assistant Professor, School of CA Journalism and Communication, The Ohio State University, Columbus Rand J. Spiro, Professor of Educational Psychology, Michigan State University, East Yael Gen, Artist, Division of Extension Lansing, MI Communications, UW–Madison
Constance Steinkuehler, Project Assistant, Sue Medaris, Artist, University Communications, UW–Madison UW–Madison
John Street, Project Assistant, UW–Madison HeeSun Park, Graduate Student Research Assistant, University of California, Santa Barbara Christopher A. Thorn, Associate Researcher, WCER, UW–Madison Jennifer Pearson, Student Help, University Communications, UW–Madison Jeffrey Watson, Project Assistant, UW– Madison Darrell Schulte, Associate Information Processing Consultant, UW–Madison Interacting with Professional Audiences David J. Tenenbaum, University Relations Edward Britton, Associate Director, National Specialist, Office of News and Public Affairs, Center for Improving Science Education, UW–Madison Washington, DC Amy Toburen, Associate Director, Office of Mary Ann Huntley, Researcher, National News and Public Affairs, UW–Madison Center for Improving Science Education, Washington, DC Susan Trebach, Executive Director for Public Affairs, University of Illinois, Chicago Susan Mundry, National Center for Improving Science Education, Andover, MA Tom Wiggins, Graduate Student Research Assistant, University of California, Santa Barbara Senta Raizen, Director, National Center for Improving Science Education, Washington, Cognitive Studies of Interdisciplinary DC Collaboration
Communicating with Mass Audiences Sharon J. Derry, Professor of Educational Psychology, UW–Madison Terrance R. Devitt, Senior University Relations Specialist, Office of News and Lori DuRussell, Project Assistant, Education, Public Affairs, UW–Madison UW–Madison
81 Laura Lee Gance, Research Specialist, UW– Nualwan Narong, Graduate Student, Industrial Madison Engineering, UW–Madison
Jan O’Neill, Educational Consultant, Quality Christopher Thorn, Associate Researcher, WCER, Leadership by Design (QLD), Madison, WI UW–Madison
Mark Schlager, Director, Teacher Professional Enlie Wang, Project Assistant, UW–Madison Development, SRI International, Menlo Park, CA Jefferey Watson, Project Assistant, UW–Madison
Information Resource Coordination Norman Webb, Senior Research Scientist, WCER, UW–Madison Armand Ardika, Student Help, UW–Madison PaulaA.White,ProjectManager,NISE,UW– ConneMara Bazley, Teacher, Roaring Fork Madison High School, Carbondale, CO Austen Zuege, Undergraduate Student, Industrial Barrett S. Caldwell, Associate Professor of Engineering, UW–Madison Industrial Engineering, Purdue, West Lafayette, IN Formative Evaluation
Nicole Canty, Program Assistant, School of Dianne Bowcock, LEAD Center Researcher, Engineering, UW–Madison UW–Madison
Calvin Chan, Graduate Student, UW–Madison Dianne C. Bowcock, Assistant Scientist, LEAD Center, UW–Madison William H. Clune, William Voss-Bascom Professor Emeritus of Law, UW–Madison Sue Daffinrud, Research Specialist, LEAD Center, UW–Madison Sharon J. Derry, Professor of Educational Psychology, UW–Madison Ramona L. Gunter, Research Training Program Fellow, School of Education, UW–Madison Virginia Emery, Student Hourly, UW– Madison Olga Lucia Herrera, Research Specialist, UW– Madison Christopher Hoadley, Research Scientist, Center for Technology in Learning, SRI Sarah Mason, Associate Researcher, WCER, International, Palo Alto, CA UW–Madison
Christine Lee, Researcher, UW–Madison Susan Millar, Director, LEAD Center, UW– Madison Susan Loucks-Horsley, Senior Research Associate, National Center for Improving Sarah K. A. Pfatteicher, Assistant Scientist, Science Education, Tucson, AZ (deceased) Academic Affairs, UW–Madison
82 Ann Burda Walker, Assistant Professor, Terrence S. Millar, Professor of Mathematics and Teaching and Learning, University of North Associate Dean, Graduate School, UW–Madison Dakota, Grand Forks Leonard Springer, Researcher, UW–Madison National Partnership in Advanced Computational Infrastructure NISE Leadership and Administration Supercomputer Project1 Catrina Ahlbach, Administrative Assistant, NISE, Baine Alexander, Researcher, Engineering, UW–Madison UW–Madison Barrett Caldwell, Associate Professor of Industrial Julie Foertsch, Associate Researcher, Engineering, UW–Madison Engineering, UW–Madison Ji-Eun Kim, Graphic Artist, UW–Madison Mike Litzkow, Associate Researcher, Computer Science, UW–Madison Lynn Lunde, Administrative Assistant, UW– Madison Susan Millar, Director, LEAD Center, UW– Madison Robert Mathieu, Professor of Astronomy, UW– Madison Greg Moses, Associate Dean, Engineering, UW–Madison Lois Opalewski, Administrative Assistant, UW– Madison Graduate Science, Mathematics, Engineering, and Technology Education2 Andrew C. Porter, Professor of Educational Psychology and Director, NISE, UW–Madison Clifton Conrad, Professor of Educational Administration, UW–Madison Ingrid Rosemeyer, Administrative Assistant, NISE, UW–Madison Christine Golde, Assistant Professor of Educational Administration, UW–Madison Deborah Stewart, Senior Editor, WCER, UW– Madison Ramona L. Gunter, Research Training Program Fellow, School of Education, UW– PaulaA.White,ProjectManager,NISE,UW– Madison Madison
Sarah Mason, Associate Researcher, WCER, UW–Madison
1 The National Partnership in Advanced Computational Infrastructure Supercomputer Project was active in Year 4 2 The Graduate Science, Mathematics, Engineering, and Technology Education was active in Years 3 and 4
83 Advisory Team
Vicki M. Bier, Associate Professor of Pat Rossman, Teacher, Elvehjem Elementary Industrial Engineering and Nuclear School, McFarland, WI Engineering, UW–Madison Leona Schauble, Professor of Educational Michael Corradini, Associate Dean, College Psychology, UW–Madison of Engineering, UW–Madison Walter Secada, Professor of Curriculum and Robert Fitch, Senior Vice President (Ret.), Instruction, UW–Madison Johnson’s Wax, Racine, WI Uri Treisman, Professor of Mathematics and LeRoy Lee, Retired Executive Director, Director, Charles A. Dana Center for Mathematics Wisconsin Academy of Sciences, Arts, & and Science Education, University of Texas– Letters, Madison Austin
Thomas A. Romberg, Sears Roebuck Foundation-Bascom Professor in Education (Mathematics Education), UW–Madison
NISE Fellows
Clifford Adelman, Senior Research Analyst, Hugh Burkhardt, Professor, Shell Centre for U.S. Department of Education/OERI, Mathematics Education, Nottingham, UK Washington, DC Judith Burry-Stock, Professor, Program of Jean-Pierre Bayard, Professor of Electrical Educational Research, Area of Professional and Electronic Engineering, California State Studies, University of Alabama, Tuscaloosa University at Sacramento Jeanne Rose Century, Senior Research Associate, Craig Bowen, Assistant Professor of Center for Science Education, Newton, MA Chemistry and Biochemistry, University of Southern Mississippi, Hattiesberg Kathy Comfort, Educational Consultant, California Department of Education, Sacramento Lia V. Brillhart, Coordinator of Engineering, Triton College, River Grove, IL Norma Davila, Education Coordinator, Puerto Rico Systemic Initiative, San Juan Cheryl Brown-Kovacic, Mathematics and Science Assessment Coordinator, Hubert Dyasi, Director, City College of New Albuquerque Public Schools, Albuquerque, York NM Stephen Ehrmann, Vice President of the Charles Bruckerhoff, Professor of Science Teaching, Learning, and Technology Affiliate of Education, University of Connecticut, Chaplin the American Association for Higher Education,
84 and Director of the Flashlight Program, Susan Millar, Director, Center for Learning Washington, DC through Evaluation, Adaptation and Dissemination, UW–Madison Susan Friel, Associate Professor, University of North Carolina–Chapel Hill Joel Mintzes, Professor of Biology, University of North Carolina at Wilmington Uwe Hilgert, Adjunct Instructor, University of Arizona, Tucson Marco Molinaro, Director of Multimedia Development for the Lawrence Hall of Science, Andrew Jackman, Associate Dean of Science, University of California, Berkeley Mount Hood Community College, Portland, OR Judith Mumme, Director, Mathematics Renaissance Program, Camarillo, CA John Jungck, Professor of Biology, Beloit College, and Director of the BioQUEST Angela O’Donnell, Associate Professor, project, Beloit, WI Department of Educational Psychology, Rutgers University, New Brunswick, NJ Jane Butler Kahle, Professor, Miami University, Oxford, OH Chris Ohana, Assistant Professor of Elementary Education, Western Washington University, Okhee Lee, Assistant Professor, University of Bellingham Miami, Coral Gables, FL James Ridgway, Professor, Lancaster University, Paul LeMahieu, Professor, Delaware Lancaster, UK Education Research and Development Center, University of Delaware, Newark Elaine Seymour, Director of Ethnography and Evaluation Research, University of Colorado- Ned Levine, Principal, Willson Science and Boulder Technology School, Bozeman, MT Marcelle Siegel, Postdoctoral Researcher, NISE Eileen Lewis, Professor of Chemistry at Secondary Teacher Education Project, UW– Cañada College, and Director of Assessment Madison and Evaluation for the ModularCHEM Consortium at the University of California, Edward Silver, Senior Scientist, University of Berkeley Pittsburgh
Sharon Lynch, Associate Professor, George Margaret Schwan Smith, Research Associate, Washington University, Washington, DC University of Pittsburgh
Flora McMartin, Assessment Director, Cary Sneider, Director, Astro/Physics Education, National Engineering Educational Delivery University of California, Berkeley System, University of California, Berkeley Barbara Spector, Professor of Science Education, University of South Florida
85 Leonard Springer, Researcher, UW–Madison Karen Worth, Senior Scientist, Education Development Center, Newton, MA Mary Kay Stein, Research Associate, University of Pittsburgh Michael Zeilik, Professor of Physics and Astron- omy, University of New Mexico, Albuquerque Sheila Tobias, Author/Educator, Tucson, AZ Susan Zeyher, Research Specialist, NISE Uri Treisman, Director, Dana Center for Information Resource Coordination Team, UW– Mathematics, University of Texas-Austin Madison
Collaborating Organization Representatives
Martin Apple, Executive Director, Council of Gary Marx, Executive Director, American Scientific Society Presidents Association of School Administrators
Robert Barkley, Manager, National Center for Michael Neuschatz, Senior Research Associate, Innovation, National Education Association American Institute of Physics
John Barth, Director of Education Policy, John Rigden, Director of Physics Programs, National Governors’ Association American Institute of Physics
Jerry Bell, Program Director, American Linda Rosen, Executive Director, National Association for the Advancement of Science Council of Teachers of Mathematics
Rolf Blank, Director of Education Indicators, Shirley Schwartz, Director of Special Projects, Council of Chief State School Officers Council of the Great City Schools
Thomas Corcoran, Senior Research Associate, Marcia Sword, Executive Director, Mathematical Consortium for Policy Research in Education Association of America
Milton Goldberg, Senior Vice President of Susan Traiman, Director of Education Initiatives, Education, National Alliance of Business Business Roundtable
Pam Magasich, Assistant for Accreditation, Sylvia Ware, Director, Education Division, National Council for Accreditation of Teacher American Chemical Society Education Ruth Wattenberg, Director of Educational Issues, James Mahoney, Director, Academic Student American Federation of Teachers and International Services, American Association of Community Colleges Gerry Wheeler, Executive Director, National Science Teachers Association
86 National Advisory Board
Edward F. Ahnert, President, Exxon Wilfred A. Kenney, Jr., Program Manager, Education Foundation, Irving, TX Western Hemisphere Integrated Supply, Xerox Corporation, Webster, NY Anna Caroline Ball, President and CEO, Ball Horticultural Company, West Chicago, IL Marcia Linn, Professor and Chair, Cognition and Development Area, University of California, George R. Boggs, President, Palomar Berkeley Community College, San Marcos, CA Gene Maeroff, Director, Hechinger Institute on Tony Bryk, Professor of Education and Education and the Media, Teachers College, Sociology, University of Chicago Columbia University, New York
Goéry Delacôte, Executive Director, The Dean Nafziger, Sylvan Learning Systems, Inc, Exploratorium, San Francisco, CA Baltimore, MD
Eugene M. DeLoatch, Dean, School of John W. Porter, Chief Executive Officer, Urban Engineering, Morgan State University, Education Alliance, Ann Arbor, MI Baltimore, MD F. James Rutherford, Chief Education Officer, Susan Fuhrman, Dean and Professor of American Association for the Advancement of Education, Graduate School of Education, Science, Washington, DC University of Pennsylvania, Philadelphia Bonita Talbot-Wylie, Teacher, Excelsior Manuel Gomez, Professor of Physics and Elementary School, Shorewood, MN Director, Resource Center for Science and Engineering, University of Puerto Rico, San William Vélez, Professor of Mathematics, Juan University of Arizona, Tucson
Susan L. Graham, Professor of Computer Science, University of California, Berkeley
87 Appendix B NISE Conferences, Forums, and Workshops
Year One
Conference Title Team Location Date
College Level One Planning CL-1 Madison, WI June 23-24, 1995 Workshop: Articulation, Equity, and Literacy Issues
Systemic Evaluation Conference SESR Madison, WI January 4-5, 1996
First Annual Forum PD/IPA Arlington, VA March 18-19, 1996 Professional Development for Science and Mathematics Education: Putting Knowledge into Action
Year Two
Policy Analysis of Systemic Reform PASR Madison, WI July 25-26, 1996 Conference
Understanding Interdisciplinary CSIC Madison, WI Nov. 15-16, 1996 Teamwork Symposium
Systemic Reform Seminar I PASR Madison, WI January 15, 1997
Systemic Reform Seminar II PASR Madison, WI January 17, 1997
Systemic Reform Seminar III PASR Madison, WI February 4, 1997
Systemic Reform Seminar IV PASR Madison, WI February 14, 1997
Second Annual Forum PASR/ Washington, DC February 24-25, 1997 Research on Systemic Reform: What SESR/ Have We Learned? What Do IPA/FE We Need to Know?
Evaluating Systemic Initiatives SESR Madison, WI March 13-14, 1997 Conference
Cooperative Learning in Higher CL-1 Madison, WI May 16-17, 1997 Education
89 Year Three
National Education, Outreach, and NPACI Madison, WI September 11-13, 1997 Training Planning/Evaluation Workshop
Team Leader Team Retreat TLT Madison, WI September 15, 1997
Special Emphasis Panel Meeting SR Arlington, VA October 6-7, 1997
NISE National Advisory Board NAB Madison, WI October 14, 1997 Meeting
Professional Development Team PD Washington, DC November 14-15, 1997 Conference
Graduate Education Planning GE Madison, WI December 8-9, 1997 Conference
Third Annual Forum CL-1/ Washington, DC February 23-24, 1998 Effective Practices and Their IPA/FE Indicators in Reforming Postsecondary SMET Courses: Shapes of the Future
Professional Development Miniforum PD Las Vegas, NV April 17, 1998
Special Emphasis Panel Meeting SR Arlington, VA June 8-9, 1998
Graduate Education Forum GE Arlington, VA June 29-30, 1998
Year Four
NISE National Advisory Board Meeting NAB Madison, WI October 26, 1998
Fourth Annual Forum SR Arlington, VA February 1-2, 1999 Evaluation of Systemic Reform in Mathematics and Science
College Level One Institute for CL-1 Madison, WI April 1-3, 1999 Technology Planning Retreat
Year Five
Team Leader Team Retreat TLT Madison, WI August 23-24, 1999
90 NISE National Advisory Board NAB Madison, WI October 18, 1999 Meeting
National Science Teachers Association PD Orlando, FL April 7, 1999 Conference Reforming Professional Development
Fifth Annual Forum SR Detroit, MI May 22-23, 2000 Diversity and Equity Issues in Mathematics and Science Reform: What Do We Know? What Do We Need to Know?
91 92 Appendix C NISE Publications
Research Monographs
Item Code Author Title
RM1 Knapp, M. Between systemic reforms and the mathematics and science classroom: The dynamics of innovation, implementation, and professional learning
RM2 Kirst, M., & Bird, R. The politics of developing and maintaining mathematics and science curriculum standards
RM3 Rodriguez, A. Counting the runners who don’t have shoes: Trends in student achievement in science by socioeconomic status and gender within ethnic groups
RM4 Tate, W. Race, SES, gender, and language proficiency trends in mathematics achievement: An update
RM5 O’Donnell, A., Cognitive processes in interdisciplinary groups: Problems and DuRussel, L., & possibilities Derry, S.
RM6 Webb, N. Criteria for alignment of expectations and assessments in mathematics and science education (in collaboration with the Council of Chief State School Officers)
RM7 Derry, S., DuRussel, L., Individual and distributed cognitions in interdisciplinary & O’Donnell, A. teamwork: A developing case study and emerging theory
RM8 Ridgway, J. The modeling of systems and macro-systemic change: Lessons for evaluation from epidemiology and ecology
RM9 Kahle, J. B. Reaching equity in systemic reform: How do we assess progress and problems?
RM10 Kennedy, M. Defining optimal knowledge for teaching science and mathematics
RM11 Springer, L., Stanne, Effects of small-group learning on undergraduates in science, M. E., & Donovan, S. mathematics, engineering, and technology: A meta-analysis
93 RM12 Lee, O. Current conceptions of science achievement in major reform documents and implications for equity and assessment
RM13 Kennedy, M. Form and substance in inservice teacher education
RM14 Rodriguez, A. Equity through systemic reform: The case of whole-school mathematics and science restructuring in Puerto Rico
RM15 Eveland, W. P., & A review of educational hypermedia uses and effects with Dunwoody, S. application to the study of learning via the World Wide Web
RM16 Clune, W. Toward a theory of systemic reform: The case of nine NSF Statewide Systemic Initiatives
RM17 Mundry, S., Spector, Working toward a continuum of professional learning B., Stiles, K., & experiences for teachers of science and mathematics Loucks-Horsley, S.
RM18 Webb, N. L. Alignment of science and mathematics standards and assessments in four states
RM19 Ohana, C. Preservice teacher cohorts and their implications for mathematics and science education
RM20 Heck, D. J., & Purposes and issues of systemic evaluation in education Webb, N. L.
RM21 Rodriguez, A. J. A Cross-case analysis of the Puerto Rico Statewide and the Miami-Dade Urban Systemic Initiatives: Promising examples of equity in systemic reform
Occasional Papers
Item Code Author Title
OP1 Raizen, S. Standards for science education
OP2 House, E. R. Implementing evaluation findings in government agencies
OP3 Clune, W. (Ed.) Commentaries on mathematics and science standards
OP4 Tobias, S. Some recent developments in teacher education in mathematics and science. A review and commentary
94 OP5 Kirst, M., Bird, R., & Tensions between mathematics and science content Raizen, S. A. standards and local politics
OP6 Cooper, J. & Small-group instruction: An annotated bibliography of Robinson, P. science, mathematics, engineering and technology resources in higher education
OP7 White, P. A. (Ed.) NISE Fellows program: Feedback from past Fellows
OP8 Millar, S. B. The role of formative evaluation in the development of an interdisciplinary academic center
Workshop Reports
Item Code Author Title
WR1 Bisgaard, S., et al. College Level One: Articulation, equity, and literacy issues. The report of a workshop organized by the College Level One Team
WR2 Webb, N. L., & Evaluation Strategies Project: A report of the Evaluation Heck, D. J. Questions Conference of the Evaluation Strategies Project
WR3 Hewson, P., & Professional development for science and mathematics Loucks-Horsley, S. education: Putting knowledge into action. A synopsis of the First Annual Forum
WR4 Clune, W. H., Millar, Research on systemic reform: What have we learned? What S. B., Raizen, S. A. do we need to know? Synthesis of the Second Annual NISE Webb, N. L., Britton, Forum. Volume 1: Analysis and Volume 2: Proceedings E.D.,Bowcock,D.C., Gunter, R. L., & Mesquita, R.
WR5 Gance, L. L. Understanding interdisciplinary teamwork: Challenges for research and practice
WR6 Millar, S. B. (Ed.) Indicators of success in postsecondary SMET education: Shapes of the future. Synthesis and proceedings of the Third Annual NISE Forum
WR7 Millar, T. S., et al. Synthesis of the science, mathematics, engineering and technology Graduate Education Forum
95 WR8 Webb, N. L. Evaluation of systemic reform in mathematics and science: Synthesis and proceedings of the Fourth Annual NISE Forum
Briefs
Vol. 1, Loucks-Horsley, Principles of effective professional development for No. 1 S., Stiles, K., & mathematics and science education: A synthesis of Hewson, P. standards
Vol. 1, Webb, N. Determining alignment of expectations and assessments No. 2 in mathematics and science education (in collaboration with the Council of Chief State School Officers)
Vol. 1, Weiss, I. The status of science and mathematics teaching in the No. 3 United States. Comparing teacher views and classroom practice to national standards
Vol. 2, Ridgway, J. From barrier to lever: Revising roles for assessment in No. 1 mathematics education
Vol. 2, Eveland, W., & Surfing the Web for science: Early data on the users and No. 2 Dunwoody, S. uses of The Why Files
Vol. 2, Kahle, J. B. Measuring progress toward equity in science and mathematics No. 3 education
Vol. 3, Mundry, S., & Designing professional development for science and No. 1 Loucks-Horsley, S mathematics teachers: Decision points and dilemmas
Vol.3, Kennedy, M. M. Form and substance in mathematics and science No. 2 professional development
Vol. 3, Meyer, R. H. Value-added indicators: A powerful tool for evaluating No. 3 science and mathematics programs and policies
96 Internal Documents These confidential reports are part of the NISE’s Internal Document series.
Item Code Author Title
ID1 Millar, S. B., Baseline report on the Team Leaders Team and Management Team Bowcock, D. C., & Burda, A. C.
ID2 Millar, S. B., First Annual NISE Forum Bowcock, D. C., & Burda, A. C.
ID3 Millar, S. B., Baseline report on NISE “intermediaries” Bowcock, D.C., & Burda, A. C.
ID4 Millar, S. B., Report on non-Team Leader Team/Management Team participants Burda, A. C., & Bowcock, D. C.
ID5 Bowcock, D. C., Report on perspectives of the co-directors, Team Leaders Team, and Pfatteicher, Management Team S. K. A., & Millar, S. B.
ID6 Bowcock, D. C., Report on perspectives of the NISE membership Gunter, R. L., Millar, S. B., & Pfatteicher, S. K. A.
ID7 Gunter, R. L., Second Annual NISE Forum evaluation report Mesquita, R., Bowcock, D. C., Pfatteicher, S. K. A., & Millar, S. B.
ID8 Mason, S., & Third Annual NISE Forum evaluation report Millar, S. B.
ID9 Gunter, R., & Strengthening graduate education in science and engineering: Mason, S. Promising practices and strategies for implementation
97 ID10 White, P. A. Fourth Annual NISE Forum evaluation report
ID11 White, P. A. Fifth Annual NISE Forum evaluation report
NISE Books
Clune, W. H., Osthoff, E. J., & White, P. A. (2000). Theory and practice of systemic reform of mathematics and science education. Manuscript in preparation. Derry, S. (2000). Toward a cognitive science of interdisciplinary collaboration. Manuscript in preparation. Loucks-Horsley, S., Hewson, P., Love, N., & Stiles, K. (1997). Designing professional development for teachers of science and mathematics. Thousand Oaks, CA: Corwin Press. Lynch, S. (2000). Equity and science education reform. Mahway, NJ: Erlbaum. Mintzes, J. J., Wandersee, J. H., & Novak, J. D. (Eds.). (1998). Teaching science for understanding: A human constructivist view. San Diego, CA: Academic Press. Mintzes, J. J., Wandersee, J. H., & Novak, J. D. (Eds.). (2000). Assessing science understanding: A human constructivist view. San Diego, CA: Academic Press. Mundry, S., Britton, E., Raizen, S., & Loucks-Horsley, S. (2000). Designing successful professional meetings and conferences in education: Planning, implementation, and evaluation. Thousand Oaks, CA: Corwin Press. Mundry, S., & Loucks-Horsley, S. (2000). Challenges in designing effective professional development programs. Manuscript in preparation. Mundry, S., Loucks-Horsley, S., et al. (2000). Learning to design: A field book for designing professional development for teachers of science and mathematics. Manuscript in preparation. Stiles, K., Loucks-Horsley, S., et al. (2000). Learning by design. Manuscript in preparation. Webb, N. L. (Ed.). (2000). Evaluating systemic reform in mathematics and science. Manuscript in preparation.
Joint Publications
Adelman, C. (1997). Leading, concurrent, or lagging? The knowledge content of computer science in higher education and the labor market (ED Publication No. PLLI 97-8046). Washington, DC: U.S. Department of Education.
Adelman, C. (1998). Women and men of the engineering path: A model for analysis of undergraduate careers (ED Publication No. PLLI 98-8055). Washington, DC: U.S. Department of Education.
Britton, E., Raizen, S., Kaser, J., & Porter, A. (2000). Beyond description of the problems: Directions for research on diversity and equity issues in K-12 mathematics and science education. Washington, DC: National Center for Improving Science Education.
98 NISE Web Sites
Collaborative Learning: http://www.wcer.wisc.edu/nise/cl1/CL/default.asp
College Level One: http://www.wcer.wisc.edu/nise/CL1
Field-tested Learning Assessment Guide (FLAG): http://www.wcer.wisc.edu/nise/cl1/flag/default.asp
Graduate Education Forum: http://nise.wcer.wisc.edu/gradforum/
Learning Through Technology: http://www.wcer.wisc.edu/nise/CL1/ilt/default.asp
NISE home page: http://www.nise.org
NPACI Supercomputer Project: http://www.npaci.edu/Online
NSF TechEd99 Workshop: http://www.wcer.wisc.edu/teched99/
Student Assessment of their Learning Gains (SALG): http://www.wcer.wisc.edu/salgains/instructor/
Secondary Teacher Education Project: http://www.wcer.wisc.edu/step/
The Why Files: http://whyfiles.news.wisc.edu
99 100 Appendix D NISE-Related Publications
Adelman, C. (1997). Leading, concurrent, or lagging? The knowledge content of computer science in higher education and the labor market (ED Publication No. PLLI 97-8046). Washington, DC: U.S. Department of Education. Adelman, C. (1998). Women and men of the engineering path: A model for analysis of undergraduate careers (ED Publication No. PLLI 98-8055). Washington, DC: U.S. Department of Education. Boone, W. J., & Kahle, J. B. (1998). Student perceptions of instruction, peer interest, and adult support for middle school science: Differences by race and gender. Journal of Women and Minorities in Science and Engineering, 4(4), 333-340. Bowen, C. W. (2000). A quantitative literature review of cooperative learning effects on high-school and college chemistry achievement. Journal of Chemical Education, 77(1), 116-119. Clune, W. H. (1998). The national standards in math and science: Developing consensus, unresolved issues, and unfinished business. Teachers College Record, 100(1), 6-7. Clune, W. H. (1998). The “standards wars” in perspective. Teachers College Record, 100(1), 144- 149. Clune, W. H., Osthoff, E. J., & White, P. A. (2000). Theory and practice of systemic reform of mathematics and science education. Manuscript in preparation. Clune, W. H., Porter, A. C., & Raizen, S. A. (1999, September 29). Systemic reform: What is it? How do we know? Education Week, 19(5), 31. Committee on Undergraduate Science Education, Center for Science, Mathematics, and Engineering, and Technology, National Research Council. (1999). Transforming undergraduate education in science, mathematics, engineering, and technology. Washington, DC: National Academy Press. Cooper, J., & Robinson, P. (1998). Small group instruction in science, mathematics, engineering, and technology: A discipline status report and a teaching agenda for the future. Journal of College Science Teaching, 27(6), 383-388. Derry, S. J., DuRussel, L. A., & O’Donnell, A. M. (1998). Individual and distributed cognitions in interdisciplinary teamwork: A developing case study and emerging theory. Educational Psychology Review, 10, 25-56. Derry, S. J., Gance, S. P., Gance, L. L., & Schlager, M. (2000). Toward assessment of knowledge building practices in technology-mediated work group interactions. To appear in S. Lajoie (Ed.), Computers as cognitive tools II: No more walls. Mahwah, NJ: Erlbaum Derry, S. J., & Gernsbacher, M. (Eds.). (2000). The problems and promises of interdisciplinary scholarship: Perspectives from cognitive science. (University of Wisconsin–Madison). Manuscript in preparation. DuRussel, L. A., & Derry, S. J. (1996). Sociocultural approaches to analyzing cognitive development in interdisciplinary teams. In Proceedings of the Eighteenth Annual Meeting of the Cognitive Science Society (pp. 529-533). Mahwah, NJ: Erlbaum. DuRussel, L. A., & Derry, S. J. (1998). Analogical reasoning in a natural working group. Proceedings of the Twentieth Annual Meeting of the Cognitive Science Society.
101 DuRussel, L. A., & Derry, S. J. (2000). Mental models of teamwork. (University of Wisconsin– Madison). Manuscript in preparation. Eisenhower National Clearinghouse. (1998). Ideas that work: Mathematics professional development. Columbus, OH: Author. (Digest of math portion of Loucks-Horsley, Hewson, Love, & Stiles, 1997) Eisenhower National Clearinghouse. (1999). Ideas that work: Science professional development. Columbus, OH: Author. (Digest of science portion of Loucks-Horsley, Hewson, Love, & Stiles, 1997) Eveland, W. P., Jr., & Dunwoody, S. (1997, August). Applying research on the uses and effects of hypermedia to the study of the World Wide Web. Paper presented to the Communication Technology & Policy division at the annual meeting of the AEJMC, Chicago. Eveland, W. P., Jr., & Dunwoody, S. (1997, October). Communicating science to the public via “The Why Files” World Wide Web site. Paper presented to the International Conference on the Public Understanding of Science and Technology, Chicago. Eveland, W. P., Jr., & Dunwoody, S. (1998). Users and navigation patterns of a science World Wide Web site for the public. Public Understanding of Science, 7, 285-311. Eveland, W. P., Jr., & Dunwoody, S. (1999, August). Examining information processing on the World Wide Web using think-aloud protocols. Paper presented to the Communication Technology & Policy division at the annual meeting of the AEJMC, New Orleans, LA. Eveland, W. P., Jr., & Dunwoody, S. (2000). Examining information processing on the World Wide Web using think-aloud protocols. Media Psychology, 2, 219-244. Eveland, W. P., Jr., & Dunwoody, S. (2000, June). A test of competing hypotheses about the impact of the World Wide Web versus traditional print media on learning. Paper presented to the Information Systems division at the annual meeting of International Communication Association, Acapulco, Mexico. Eveland, W. P., Jr., & Dunwoody, S. (in press). Applying research on the uses and cognitive effects of hypermedia to the study of the World Wide Web. Communication Yearbook #25. Eveland, W. P., Jr., & Dunwoody, S. (2000). A test of competing hypotheses about the impact of the World Wide Web versus traditional print media on learning. Manuscript submitted for publication. Eveland, W. P., Jr., & Dunwoody, S. (2000). An investigation of the cognitive mediators and moderators of learning from the Web versus print. Manuscript submitted for publication. Foertsch, J. (2000). Models for undergraduate instruction: The potential of modeling and visualization technology in science and math education. In Targeting curricular change: Reform in undergraduate education in science, math, engineering, and technology (pp. 37- 40). Washington, DC: American Association for Higher Education. Gamoran, A., Porter, A. C., Smithson, J., & White, P. A. (1997). Upgrading high school mathematics instruction: Improving learning opportunities for low-achieving, low-income youth. Educational Evaluation and Policy Analysis, 19(4), 325-338. Gernsbacher, M. A., & Derry, S. J. (Eds.). (1998). Proceedings of the twentieth annual meeting of the Cognitive Science Society. Mahwah, NJ: Erlbaum. Haimo, D. T. (1998). Are the NCTM standards suitable for systemic adoption? Teachers College Record, 100(1), 45-65. Heck, D. J. (1998). Evaluating equity in statewide systemic initiatives: Asking the right questions. Journal of Women and Minorities in Science and Engineering, 4(2,3), 161-181.
102 Horsley, D. L., & Loucks-Horsley, S. (1998). CBAM brings order to the tornado of change. Journal of Staff Development, 19(4),17-20. Kahle, J. B. (1997). Systemic reform: Challenges and changes. Science Educator, 6(1), 1-6. Kahle, J. B., & Rogg, S. R. (1997). Assessing systemic change: Ohio's Statewide Systemic Initiative, Discovery. Systemic Initiatives, 1(3), 10-11. Available: http://www.ehr.nsf.gov/ehr/esr/ sysinit/news.htm Kahle, J. B. (1998). Equitable systemic reform in science and mathematics: Assessing progress. Journal of Women and Minorities in Science and Engineering, 4(2,3), 91-112. Kirst, M. W., & Bird, R. L. (1997). The politics of developing and sustaining mathematics and science curriculum content standards. Advances in Educational Administration, 5, 107-132. Knapp, M. S. (1997). Between systemic reforms and the mathematics and science classroom: The dynamics of innovation, implementation, and professional learning. Review of Educational Research, 67(2), 227-266. Lee, O. (1999). Equity implications based on the conceptions of science achievement in major reform documents. Review of Educational Research, 69(1), 83-115. Lee, O., & Fradd, S. H. (1988). Science for all, including students from non-English language backgrounds. Educational Researcher, 27(4), 12-21. Lee, O., & Paik, S. (2000). Conceptions of science achievement in major reform documents. School Science and Mathematics, 100(1), 16-26. Loucks-Horsley, S. (1998). The role of teaching and learning in systemic reform: A focus on professional development. Science Educator, 7(1), 1-6. Loucks-Horsley, S. (1998). JSD forum: I have changed my emphasis. Journal of Staff Development, 19(3), 7-8. Loucks-Horsley, S., Bybee, R.W. (1998). Implementing the National Science Education Standards. The Science Teacher, 65(6), 22-25. Lynch, S. (2000). Equity and science education reform. Mahway, NJ: Erlbaum. Mathieu, R. (2000). Assessment tools to drive learning: FLAG, SALG, and other proven assessments available online. In Targeting curricular change: Reform in undergraduate education in science, math, engineering, and technology (pp. 26-31). Washington, DC: American Association for Higher Education. Millar, S. (2000). Opening doors to new spaces: The processes of implementing education reform. In Targeting curricular change: Reform in undergraduate education in science, math, engineering, and technology (pp. 41-45). Washington, DC: American Association for Higher Education. Millar, T., & Porter, A. (1998, January-March). Harnessing the Web for educational impact: The science behind the news. Envision, 13(1), 20-21. Mintzes, J. J., Wandersee, J. H., & Novak, J. D. (Eds.). (1998). Teaching science for understanding: A human constructivist view. San Diego, CA: Academic Press. Mintzes, J. J., Wandersee, J. H., & Novak, J. D. (Eds.). (2000). Assessing science understanding: A human constructivist view. San Diego, CA: Academic Press. Mundry, S., Britton, E., Raizen, S., & Loucks-Horsley, S. (2000). Designing successful professional meetings and conferences in education: Planning, implementation, and evaluation. Thousand Oaks, CA: Corwin Press.
103 Mundry, S., Spector, B., Loucks-Horsley, S., & Morrison, N. (1999, December 1). From novice to pro: Improving science and mathematics teaching by improving teacher learning. Education Week, 19(14), 21. Mundry, S., Spector, B., Loucks-Horsley, S., & Morrison, N. (2000, January-February). From novice to pro: Improving science and mathematics teaching by improving teacher learning. NSTA Reports, 11(4), 53. National Institute for Science Education. (1998). Descriptions of programs and strategies for change: Strengthening graduate education in science and engineering (Promising practices and strategies for implementation from the Graduate Education Forum). Madison: University of Wisconsin-Madison, Author. Paik, S., & Lee, O. (1998). Analysis of the conceptions of science achievement in major reform documents in the United States and Korea. Journal of the Korean Association for Research in Science Education, 18(4), 571-587. Paik, S., & Lee, O. (in press). Science achievement: Synthesis of current conceptions in major reform documents in the United States and Korea. Elementary Science Education in South Korea. Porter, A. C. (1994). National standards and school improvement in the 1990s: Issues and promise. American Journal of Education, 102(4), 421-449. Porter, A. C. (1995). The uses and misuses of opportunity-to-learn standards. Educational Researcher, 24(1), 21-27. Porter, A. C. (1998a). Dilemmas in assessing academic achievement. In N. M. Lambert & B. L. McCombs (eds.), How students learn: Reforming schools through learner-centered education (pp. 339-350). Washington, DC: American Psychological Association. Porter, A. C. (1998b). The effects of upgrading policies on high school mathematics and science. In D. Ravitch (Ed.), Brookings papers on education policy (pp. 123-172).Washington, DC: Brookings Institution Press. Porter, A. C., Youngs, P., & Odden, A. (in press). Advances in teacher assessments and their uses. In V. Richardson (Ed.), Handbook of research on teaching (4th edition). Raizen, S. (1998). Standards for science education. Teachers College Record, 100(1), 66-121. Rodriguez, A. J. (1998). Busting open the meritocracy myth: Rethinking equity and student achievement in science education. Journal of Women and Minorities in Science and Engineering, 4(2,3), 195-216. Roitman, J. (1999). A mathematician looks at national standards. Teachers College Record, 100(1), 22-44. Romberg, T. A. (1998). Comments: NCTM's curriculum and evaluation standards. Teachers College Record, 100(1), 8-21. Springer, L. (1998). Research on cooperative learning in college science, mathematics, engineering, and technology. Cooperative Learning and College Teaching, 8(3), 2-4. Springer, L., Stanne, M. E., & Donovan, S. S. (1999). Effects of small-group learning on undergraduates in science, mathematics, engineering, and technology: A meta-analysis. Review of Educational Research, 69(1), 21-51. Stein, M. K., & Mundry, S. (1999, October). Professional development for science and mathematics teachers: Dilemmas of design. The High School Principal Magazine, 7(2), 14-19.
104 Stein, M. K., Smith, M. S., & Silver, E. A. (1999). The development of professional developers: Learning to assist teachers in new settings in new ways. Harvard Educational Review, 69(3), 237-269. Stiles, K. E., & Loucks-Horsley, S. (1998). Professional development strategies. The Science Teacher, 65(6), 46-49. Tate, W. F. (1997). Race-ethnicity, SES, gender, and language proficiency trends in mathematics achievement: An update. Journal for Research in Mathematics Education, 28(6), 652-679. Tenenbaum, D. (1997, Summer). Cyber scribe tells all: Writing for the web. SEJournal [On-line]. Available: http://www.sej.org/sejournal/sej_su97.html Thorn, C. A. (2000). Knowledge management for educational information systems: What is the state of the field? (Wisconsin Center for Education Research, University of Wisconsin–Madison). Manuscript in preparation. Tobias, S. (1999). Some recent developments in teacher education in mathematics and science. A review and commentary. Journal of Science Education and Technology, 8(1), 21-31. Webb, N. L. (1998). Conditional equity metrics as tools for evaluating equity in schools and education systems. Journal of Women and Minorities in Science and Engineering, 4(2,3), 141-160. Webb, N. L. (1998b). Judging alignment of assessments and expectations in a mathematics education system. Paper presented at a conference on the quality aspects of mathematics and science education held at the Abo Akademi University, Vasa, Finland. Webb, N. L. (1998c). Thoughts on assessment in the mathematics classroom. In G. W. Bright & J. N. Joyner (Eds.), Classroom assessment in mathematics: Views from a National Science Foundation working conference (pp. 101-114). Lanham, MD: University Press of America. Webb, N. L. (1999a). Alignment of mathematics tests and mathematics reform. Mathematicians and Education Reform Forum Newsletter, 11(3). University of Illinois at Chicago. Webb, N. L. (1999b). Judging alignment of assessments and expectations in a mathematics education system. In O. Björkqvist (Ed.), Quality aspects of mathematics and science education. Reports from the Faculty of Education, No. 5. Vasa, Finland: Åbo Akademi University. Webb, N. L. (1999c). Professional development for improving assessment in mathematics and science. Paper presented at the conference Weaving Together the Strands of Systemic Reform: Professional Development, Assessment, and Research, Pretoria, South Africa, cosponsored by the South Africa National Research Foundation and the U.S. National Science Foundation, October 27-29. Webb, N. L. (1999). Tools for understanding systemic reform: Logic-of-action and theory-of-change models. Paper presented at the American Educational Research Association annual meeting, Montreal, Quebec, Canada. Webb, N. L., Century, J. R., Dávila, N., Heck, D., & Osthoff, E. (2000). Evaluation of systemic reform in mathematics and science. Madison, WI: Wisconsin Center for Education Research, University of Wisconsin. Manuscript in preparation. Webb, N. L., & Smithson, J. (1999). Alignment between standards and assessments in mathematics for grade 8 in one state. Paper presented at the American Educational Research Association annual meeting, Montreal, Quebec, Canada.
105 Wiese, D., Seymour, E., & Hunter, A. (1999, June). Report on a panel testing of the Student Assessment of their Learning Gains Instrument by faculty using modular methods to teach undergraduate chemistry. Boulder: University of Colorado, Bureau of Sociological Research. Wright, J. C., & Wright, C. S. (1998). A commentary on the profound changes envisioned by the national science standards. Teachers College Record, 100(1), 122-143. Wright, J. C., Millar, S. B., Kosciuk, S. A., Penberthy, D. L., Williams, P. H., & Wampold, B. E. (1998). A novel strategy for assessing the effects of curriculum reform on student competence. Journal of Chemical Education, 75, 986-992.
106