Editors Daniel C. Edelson A ROAD MAP Richard J. Shavelson Jill A. Wertheim FOR 21st CENTURY GEOGRAPHY EDUCATION Assessment

Recommendations and Guidelines for Assessment in Geography Education

A Report from the Assessment Committee of the Road Map for 21st Century Geography Education Project Road Map for 21st Century Geography Education Project

Assessment

Recommendations and Guidelines for Assessment in Geography Education

Editors

Daniel C. Edelson, Richard J. Shavelson, Jill A. Wertheim

National Geographic Society Washington, DC

A Report from the Assessment Committee of the Road Map for 21st Century Geography Education Project

2 of 75 This report was created by the Road Map for 21st Century The National Geographic Society is one of the world’s largest nonprofit scientific and Geography Education Project. educational organizations. Founded in 1888 to “increase and diffuse geographic knowledge,” the Society’s mission is to inspire people to care about the planet. It reaches more than 400 Road Map for 21st Century Geography Education Project million people worldwide each month through its official journal, National Geographic, and other magazines; National Geographic Channel; television documentaries; music; radio; Daniel C. Edelson, Principal Investigator films; books; DVDs; maps; exhibitions; live events; school publishing programs; interactive Virginia M. Pitts, Project Director media; and merchandise. National Geographic has funded more than 10,000 scientific research, conservation and exploration projects and supports an education program promoting The Road Map for 21st Century Geography Education Project is a collaboration between the geographic literacy. National Geographic Society, the Association of American Geographers, the National Council for Geographic Education, and the American Geographical Society. The views expressed in the report The Association of American Geographers (AAG) is a nonprofit scientific, research, and are those of the authors and do not necessarily reflect the views of these organizations. educational society founded in 1904. Its 11,000 members from more than 60 countries share interests in the theory, methods, and practice of geography (including GIScience, geographic The Road Map for 21st Century Geography Education Project has been supported in part by education, and geographic technologies). The AAG pursues its mission through its many the National Science Foundation under Grant No. DRL-1049437. Any opinions, findings, and conferences, scholarly publications, research projects, educational programs, topical specialty conclusions or recommendations expressed in this material are those of the authors and do not groups, and its extensive international network of colleagues and organizational partnerships, necessarily reflect the views of the National Science Foundation. which encompass professionals working across public, private, and academic sectors all around the world.

The National Council for Geographic Education (NCGE) works to enhance the quality, quantity, and status of geography teaching and learning in primary, secondary, university, and informal educational settings. It develops and promotes curricular materials and two journals, fosters best practices in pedagogy and geotechnology, connects educators through online communication and through its annual conference, supports research in geographic education, recognizes exceptional supporters and teachers of geography, and collaborates with other organizations that have similar goals.

The American Geographical Society is an organization of professional geographers and other devotees of geography who share a fascination with the subject and a recognition of its importance. Most Fellows of the Society are Americans, but among them have always been Suggested citation: a significant number of Fellows from around the world. The Society encourages activities Edelson, D. C., Shavelson, R. J., & Wertheim, J. A. (Eds.). (2013). A road map for 21st century that expand geographical knowledge, and it has a well-earned reputation for presenting and geography education: Assessment (A report from the Assessment Committee of the Road Map for interpreting that knowledge so that it can be understood and used not just by geographers 21st Century Geography Education Project). Washington, DC: National Geographic Society. but by others as well—especially policy makers. It is the oldest nationwide geographical organization in the United States. Its priorities and programs have constantly evolved with Information about the project and copies of reports are available at http://natgeoed.org/roadmap. the times, but the Society’s tradition of service to the U.S. government, business community, Copyright © 2013 by the National Geographic Society. All rights reserved. and nation-at-large has continued unchanged.

3 of 75 Assessment Committee

Daniel C. Edelson, Chair

Richard J. Shavelson, Co-Chair

Jill A. Wertheim, Research Director

Barbara Hildebrant, Educational Testing Service

Elizabeth R. Hinde, Arizona State University

Marianne Kenney, Denver Public Schools

Bob Kolvoord, James Madison University

David A. Lanegran, Macalester College

Jody Smothers Marcello, Sitka High School

Robert W. Morrill, Virginia Tech

Maria Ruiz-Primo, University of Colorado Denver

Peter Seixas, University of British Columbia

4 of 75 Project Steering Committee Project Advisory Board

Daniel C. Edelson, Principal Investigator and Gilbert M. Grosvenor, Chair Chair, Assessment Committee National Geographic Society

Virginia M. Pitts, Project Director

Susan Heffron, Association of American Geographers and Harm de Blij, Michigan State University Co-Chair, Geography Education Research Committee Richard G. Boehm, Texas State University-San Marcos Doug Richardson Association of American Geographers Barbara Chow, The William and Flora Hewlett Foundation

Jerome E. Dobson, American Geographical Society and Jack Dangermond, Esri Study Director, Public Understanding and Values Study Michael F. Goodchild, University of California, Santa Barbara

Kristin Alvarez Brian McClendon, Google National Council for Geographic Education Alexander B. Murphy, University of Oregon Joseph Kerski National Council for Geographic Education M. Duane Nellis, University of Idaho

Robert Dulli Trevor Packer, The College Board National Geographic Society Lee Schwartz Sarah Witham Bednarz, Chair Geography Education Research Committee

Emily M. Schell, Chair Instructional Materials and Professional Development Committee

Kathleen J. Roth, Co-Chair Instructional Materials and Professional Development Committee

Richard J. Shavelson, Co-Chair Assessment Committee

5 of 75 Table of Contents Chapter 4: A 21st Century Assessment Framework for the Geographical Sciences ...... 46 Executive Summary ...... 7 A 21st Century Assessment Framework for the Geographical Sciences Preface ...... 11 !e Content Dimension in AFGS21 Chapter 1: Context and Goals for the Road Map for !e Cognitive Dimension in AFGS21 21st Century Geography Education Project ...... 17 Using AFGS21 to Develop Speci#c Assessment Frameworks !e State of Geography Education in the United States Implications for Improving Geography Education !e Importance of Geography Education !e Need for a “Road Map” for Geography Education Chapter 5: An Example Geography Assessment Framework ...... 56 Establishing a Destination: Goals for K–12 Geography Education Assembling the Framework Development Team Describing the Destination: E"ectiveness and Balance Preparing the Team Mapping a Bright Future Developing the Framework A Road Map for Geography Assessments Conclusion

Chapter 2: Improving Teaching and Learning !rough Assessment ... 28 Chapter 6: Recommendations ...... 65

Uses of Assessment Information in Education A Focus on Assessment in E"orts to Improve Geography Education Considerations for the Design of Assessments A New Approach to the Assessment of Geography !e Role of Frameworks in the Design of Assessments Shared Frameworks and Assessments Implications for Improving Geography Teaching and Learning Capacity Building Knowledge Base Chapter 3: !e Current Context for K‒12 Geography Assessment ..... 38

Overview of Current Geography Frameworks Appendix A: Additional Details About the Assessment Study ...... 67 Current Geography Assessment Practices Appendix B: Example Assessment Framework ...... 71 Findings Appendix C: Assessment Committee Biographies ...... 72 Implications for Improving Geography Education References ...... 74

In the online version of this report, the icon on the lower left corner of the following pages will bring you back to this Table of Contents.

6 of 75 Geography Executive Context and Purposes of Geography Example Appendices Preface Assessment Recommendations References Summary Goals Assessment Assessment Framework A, B, C Framework

Executive Summary enable assessment developers to address the critical prepare students for the demands of the modern world. issues in assessment design. As with the other Road Map Assessments of geographic concepts and skills expose Introduction Project reports, this one places a particular emphasis the failure of our educational system in geography, on how to move geography education toward a balance con#rming that a vast majority of American students Student assessments typically are viewed simply as between developing geographic knowledge and learning are geographically illiterate. !e 2010 National Assess- indicators of educational progress, but this report is to engage in geographic practices. Speci#cally, this ment of Educational Progress (NAEP), known as based on the recognition that the utility of assessments report follows the balanced approach to geography “!e Nation’s Report Card,” found that fewer than can extend far beyond this role in education. For education called for in the second edition of Geography 30% of American students were pro#cient in geography, example, the results of student assessments can provide for Life: National Geography Standards (He"ron & meaning that more than 70% of students at fourth, critical information for decision making in education Downs, 2012), the national standards document that eighth, and 12th grades were unable to perform at the policy and practice. In addition, what is being assessed also was developed through a collaborative e"ort of the level that is expected for their grade (National Center four Road Map Project partners. and how it is assessed becomes a means to communicate for Education Statistics, 2011). At 12th grade, more than 30% of students scored below “basic,” indicating goals and priorities to teachers, students, and other Background stakeholders in K–12 education. that they had not mastered even foundational geograph- !e four partners launched the Road Map Project ic concepts or skills. !is report explores how changes and improvements because they share a concern that the poor state of in assessment practices can support e"orts to improve The Value of Assessment for geography education in America is a threat to our K–12 geography education. !e report is one of three Improving Geography Education country’s well-being, and by extension, to the well-being reports developed as part of the National Science Foun- of the global community. !e partners share the belief dation-funded Road Map for 21st Century Geography !is report takes the position that the primary pur- that geography education is essential for preparing the Education Project pose of educational assessment should be for making , a collaboration of four national general population for careers, civic lives, and personal informed decisions. Because they typically regard assess- associations committed to improving geography educa- decision making in contemporary society. !ey also ment as a separate activity from instruction, educators, tion—the National Geographic Society, the Association believe it is essential for the preparation of specialists students, parents, and policy makers often overlook of American Geographers, the American Geographical capable of addressing critical societal issues in the areas invaluable ways assessments can support and improve Society, and the National Council for Geographic Edu- of social welfare, economic stability, environmental teaching and learning. !e report describes four ways cation. !e other two project reports focus on geogra- health, and international relations. !ey fear that by that assessment results can contribute to improvements phy education research and on instructional materials neglecting geography education today, we are placing in teaching and learning by providing evidence that and professional development for geography. the welfare of future generations at risk. guides critical decisions. !is report begins by laying out a set of issues critical While inspiring examples of highly e"ective geography 1. The results of assessments can inform teachers’ for the design of assessments that support instructional education can be found in nearly every part of the Unit- instructional decisions. When assessments are improvement and by reviewing current assessment ed States, for the overwhelming majority of students, integrated into instruction, they can improve frameworks and practices in K–12 geography education. the amount of geography instruction they receive, the its effectiveness by enabling the teaching and !e second half of the report contains a proposal for preparation of their teachers to teach geography, and the learning process to be tailored to students’ a new approach to assessment in geography that will quality of their instructional materials are inadequate to specific needs.

The Road Map Project | Assessment Report | Executive Summary 7 of 75 Geography Executive Context and Purposes of Geography Example Appendices Preface Assessment Recommendations References Summary Goals Assessment Assessment Framework A, B, C Framework

2. The results of assessments can be used to inform challenging. Four of the key decisions in the design of assessment framework also provides guidance on item decisions about students’ academic programs. assessments are: quality and cost constraints. Assessments introduced at appropriate intervals 1. Selection of goals: What are the specific content can be used to measure a student’s proficiency Because of the role assessment frameworks can play in and practices required for the competencies be- against benchmark goals for that student at that guiding the design of assessments, this report focuses on ing assessed? point in his or her academic career. The results of the development and dissemination of new assessment these assessments can be used to inform deci- 2. Item characteristics: What are the characteristics frameworks as a means to guide the development of sions about that student’s academic program. of the item that will be used to assess a high-quality assessments that evaluate 21st century competency (e.g., task type, response mode, 3. The results of assessments can be used to inform knowledge and skills. scoring system)? decisions about the function and effectiveness of educational programs. Aggregated results of 3. Item quality: How will the technical quality of Assessment in Geography Today student assessments can be used as part of pro- the item be measured (e.g., validity, reliability, gram evaluation. Used in this way, assessments fairness)? To determine how well current assessment projects can inform decisions about program selection, are aligned with the goals of geography education, as 4. Cost effectiveness: How much time and program implementation, and other aspects of described in Geography for Life, this report examines the resources are required to create, administer, and instruction. They also can be used in evaluations nature of existing assessment frameworks and current score the assessment? of the performance of classes, schools, and larger assessment practices in K–12 geography education. units that might reveal challenges that need to be In making these design decisions, assessment developers addressed; likewise, they can be used to inform Assessment Frameworks in Geography must carefully consider the nature of the content and Education Today decisions about where to focus resources. practices to be assessed, the context in which they will !ere are currently three prominent assessment 4. The results of assessments can be used to build be administered, the population whose competencies a knowledge base for future decision making. frameworks being used in K$12 geography education in will be assessed, and the purposes for which the results Assessment results used for research enable the United States: will be used. examination of broader questions than those • National Assessment of Educational Progress in Geography (1994, 2001, 2010). The NAEP geog- revealed by the performance of a specific student One way developers of assessments minimize the or program. They can be used to examine general raphy framework is the basis for assessments that challenge of addressing these considerations is through questions about teaching and learning geog- are used in a national evaluation of geography assessment frameworks. An assessment framework raphy, such as what makes one approach more education outcomes at grades 4, 8, and 12. effective than another, or how students develop plays the role of an outline in writing or a functional • Advanced Placement Human Geography spatial learning skills. The results of these stud- speci#cation in engineering. Frameworks provide (2000). The framework for Advanced Placement ies can inform efforts to improve education over guidelines for making decisions in the development Human Geography (APHG) guides the design longer time scales. of an assessment. of the examination used by the College Board to determine if high school students who have Considerations in the Design Contemporary assessment frameworks use a two- completed an AP course in human geography of Assessments dimensional framework to lay out the content and have achieved a level of mastery equivalent cognition targets for an assessment, their relative to successful completion of an undergraduate Designing accurate and useful assessments is extremely importance, and item characteristics. A comprehensive course in the subject.

The Road Map Project | Assessment Report | Executive Summary 8 of 75 Geography Executive Context and Purposes of Geography Example Appendices Preface Assessment Recommendations References Summary Goals Assessment Assessment Framework A, B, C Framework

• National Assessment of Educational Progress for that students use any geographic practices at all. Analyz- A 21st Century Assessment Science (2008). The NAEP Science framework ing geographic information was assessed in 21% of all Framework for the Geographical is the basis for assessments that are used in a large-scale items, but other geographic practices were Sciences national evaluation of science education out- rarely assessed (Figure 1). comes at grades 4, 8, and 12. It includes concepts !is report introduces a new assessment framework and practices that are included in Geography for !e study found that assessments are largely failing to serve as a blueprint that guides the development of Life, such as, Earth processes, ecology, human- to probe deep understanding. More than half of the a new generation of geography assessments. Called a environment interaction, data analysis, and com- large-scale assessment items required only declarative munication. 21st Century Assessment Framework for the Geographical knowledge (e.g., knowing that), often at the level of rec- Sciences (AFGS21), its goal is to support the design ognizing a de#nition. Only 28% assessed students’ pro- !is report concludes that these three frameworks of assessments that are aligned with the goals of the cedural knowledge (e.g., knowing how), which includes place too little emphasis on geographic practices to national geography standards. accurately assess students’ mastery of the goals outlined reading and gathering information from maps, graphs, in Geography for Life, although the NAEP Science and texts. And, only 17% of geography items required AFGS21 was designed to be a general assessment framework serves as a model of how to assess other schematic knowledge (e.g., knowing why), Figure 1. Frequency Distribution of Large-Scale Geography scienti#c practices. which includes explaining an unfamiliar Assessment Items That Target Each Geographic Practice context by drawing on general geographic Assessment Practices in Geography Today principles or models. 80% !is report includes the #ndings of a study, 70% commissioned for the report, of existing K–12 Finally, the study revealed widespread prob-

ems 60% geography assessments. !e study was conducted to lems with item quality. Of the items stud- gather in-formation about how well current assessment ied, 60% were judged to have problems that 50% practices re%ect the goals of Geography for Life, and could impede students’ ability to accurately 40% represent what they know and what they how well they implement the principles of e"ective 30%

assessment design described above. can do with their geography knowledge. entage of It 20% rc !e report’s review of current assessment !e study found the content evaluated by current Pe 10% assessments is unevenly distributed across the goals practices reveals that both assessment 0% described in Geography for Life. For example, 40% of all frameworks and actual assessments do not e s s items across both large-scale and classroom assessments re%ect the balance between assessing what non aphic aphic evaluated knowledge from only three out of the 16 students know and their ability to apply ganizing question aphic data information ommunicating their knowledge that is required to evaluate ering questionsc content standards, and far fewer items assessed content w posing geogr geogr ans aphic information from the Environment and Society category compared the development of 21st century geography analyzing geogr llecting and or with any other content area. !e study also found that competencies. Even within knowledge and co and designing solutiongeogr geographic practices are not being widely assessed. Only practices, the review of assessment items 30% of large-scale geography assessment items required reveals a large imbalance, as well. Geographic Practices

The Road Map Project | Assessment Report | Executive Summary 9 of 75 Geography Executive Context and Purposes of Geography Example Appendices Preface Assessment Recommendations References Summary Goals Assessment Assessment Framework A, B, C Framework

framework that would cover all of K–12 geography, dimension are divided into knowing and understanding !e report describes a process for developing speci#c with the idea that it will be a template for more speci#c and geographic practices. !e geographic practices, in assessment frameworks from AFGS21 that begins with assessment frameworks for speci#c contexts, audiences, turn, are divided into six categories. A central feature de#ning the subset of content and cognition to be and purposes. !e report also lays out a process for of the framework is a matrix that is used to blend the assessed, and the detailed process continues through creating speci#c assessment frameworks from AFGS21 two dimensions systematically, articulating the speci#c the stage of specifying the desired distribution and and for using those frameworks to develop assessments. performance expectations to be assessed. !e contents characteristics of items. !e assessments developed !e two dimensions of AFGS21 are designated as of a cell within the matrix might describe a geographic through this process and implemented by teachers, content and cognition. !e categories in the content concept that students would be expected to know or program and material developers, and researchers have dimension are de#ned by the content standards in understand, or a cell might refer to the application of a the potential to be powerful tools for advancing the Geography for Life. !e categories in the cognitive geographic practice using a particular concept. goals of geography education reform.

The Road Map Project | Assessment Report | Executive Summary 10 of 75 Geography Executive Context and Purposes of Geography Example Appendices Preface Assessment Recommendations References Summary Goals Assessment Assessment Framework A, B, C Framework

Preface Project Structure Once the project was funded, the project partners established a Steering Committee consisting of one !e Road Map Project was organized into four parallel Background representative of each of the partner organizations, the e"orts. !ree e"orts were headed by committees that committee chairs and co-chairs, and the project director. !is report is a product of the Road Map for 21st were tasked with creating reports that review the Century Geography Education Project. !e Road Map current status of their area of expertise and establish Report Development Process Project has its origins in a directive from Congress in recommendations for the future. !e fourth e"ort—a Each report was created by a committee convened by the 2010 budget appropriation for the National Science study of public understanding and values—developed the partners. It represents a consensus of the members Foundation (NSF) that instructed the Foundation a survey, administered it, and analyzed its results. of the committee. All three of the consensus reports to “work with external partners with experience in As part of the proposal process, the partner organiza- created by the Road Map Project were developed geographic education to improve geography teaching, tions identi#ed chairs and co-chairs for the three com- following the same process, described below. training, and research in our Nation’s schools” (U.S. mittees and a director of the study. Each of the partner House of Representatives, 2009, p. 767). organizations served as the administrative host for one Committee Formation In the spring of 2010, the National Geographic Society of the project’s four e"orts and provided professional !e committees were recruited from lists of nominees responded to this opportunity with a proposal to work sta" and administrative support for that e"ort. !e four and alternatives recommended by the Steering with three other national organizations—the Associa- e"orts, their hosts, and their chairs, as established by Committee. !e committees were constructed to have tion of American Geographers, the American Geo- the partners, were: representation from all of the following: • Instructional Materials and Professional graphical Society, and the National Council for Geo- • academic experts in geographic education, graphic Education—to create a “road map” for e"orts Development Committee Administrative host: • academic geographers, to improve geographic education. Building on three National Council for Geographic Education • academic experts in education in other areas of decades of collaboration, the partners argued that it was Chair: Emily M. Schell social studies and science, time to launch an initiative that would have large-scale Co-chair: Kathleen J. Roth impact across the United States over the course of the • Assessment Committee • K–12 practitioners (teachers and administrators), new decade. !ey proposed to undertake the Road Map Administrative host: • experts in the specific foci of each committee Project to construct plans for the initiative. !e project’s National Geographic Society (assessment, professional development, Daniel C. Edelson goal would be to learn from the lessons of earlier edu- Chair: instructional materials development, educational Co-chair: Richard J. Shavelson cational improvement e"orts in geography and other research), and • Geography Education Research Committee subjects to establish guidelines and set priorities for • perspectives from outside the United States. Administrative host: this new initiative. Association of American Geographers !e Steering Committee wrote initial charges to the Sarah Witham Bednarz Following extensive review, the National Science Chair: Co-chair: Susan Heffron committees based on the goals of the original project Foundation awarded a grant to support the Road Map • Study of Public Understanding and Values proposal. Speci#cally, the Instructional Materials Project in September 2010. Administrative host: and Professional Development Committee was American Geographical Society charged with making recommendations about the Director: Jerome E. Dobson design of instructional materials and the education

The Road Map Project | Assessment Report | Preface 11 of 75 Geography Executive Context and Purposes of Geography Example Appendices Preface Assessment Recommendations References Summary Goals Assessment Assessment Framework A, B, C Framework

of teachers. !e Assessment Committee was charged all input from other sources was treated as advisory. !e !e presenters at this workshop are listed on page 16. with developing a framework for assessing progress committees were each sta"ed by a research director with Following the workshop, as part of their work on the toward geographic literacy across the progression from a doctorate in a related #eld and provided with a budget reports, all three committees conducted additional kindergarten through high school. !e Geography to seek input from outside experts. Each committee research activities speci#c to their charge. Each of Education Research Committee was charged with met face-to-face several times over the course of the the three committees held face-to-face meetings in developing an agenda for educational research that research and writing process, in addition to conducting September 2011, January 2012, and April/May 2012 would lay out questions about learning, teaching, regular conversations via conference call. Information to work on draft-related tasks, resolve open issues, and and educational change that must be answered to sharing among committees was facilitated through regu- plan work going forward. maintain the e"ectiveness of geographic education lar conversations among research directors and chairs/ into the future. co-chairs. Review and Comment Phase Cross-committee review. First drafts of proposals !e chairs and co-chairs of the three committees, All three committees collaborated on the organization were distributed to the other committees for review together with the project director and the committee of a workshop on geographic thinking in June 2011. in January 2012. Each committee identi#ed two to research directors, formed a leadership team with the !e intent of this workshop, held in Washington, DC, #ve representatives to review the other committees’ purpose of ensuring coordination and collaboration was to bring together individuals who have insights into reports. !e intent of this review was to identify any across their committees. !e members of the leadership “expert” geographic thinking to address a series of ques- cross-committee issues that needed to be addressed, team maintained close communication with each tions related to the committees’ interests in describing and to give each committee input and feedback they other, but each committee was empowered to make its geographic literacy. Prior to the workshop, through a se- might incorporate prior to public review. Each of the own decisions and to exercise independent editorial ries of conference calls and online discussions, the com- committees then met in person or via conference call judgment over its own product. Each committee mittees identi#ed a set of core questions they sought to to discuss the feedback and determine how to received substantial input and feedback from a variety have addressed at the workshop. Questions addressed at incorporate it into their next draft. of sources. However, they were not required to obtain the workshop included approval for their products from their host organization, Public review. Revised drafts of the reports were any of the other partner organizations, the National • “How do geographers reason about space?” made available for public comment in March 2012. Science Foundation, or any other outside individual • “How do people develop spatial reasoning?” !e release of the draft reports was announced on or organization. As a result, their reports re%ect the • “How do professionals in geographic fields the project website, and announcements about the opinions and judgment of their authors. apply geography?” public review were distributed to members of the Association of American Geographers, the American Research and Draft Phase • “How do geographers frame questions and Geographical Society, the National Council for problems differently compared with other fields?” Each of the committees met for a kick-o" meeting Geographic Education, and the National Geographic in Washington, DC, in January 2011. As part of the • “How do you train geographers?” and Alliance Network. All of the presenters at the June kick-o" meeting, each committee reviewed and re#ned • “What can we learn from how other disciplines 2011 Geographic !inking Workshop, as well as others its charge. Once the committees were convened, they have characterized skills, practices, and ways of who had contributed to the work of the di"erent were given #nal editorial authority over their reports; thinking?” committees, were invited to comment as well.

The Road Map Project | Assessment Report | Preface 12 of 75 Geography Executive Context and Purposes of Geography Example Appendices Preface Assessment Recommendations References Summary Goals Assessment Assessment Framework A, B, C Framework

Review Board. In an e"ort to ensure that the #ndings and recommendations. !ey are being assisted people, places, and cultures around the world. Further, committees would obtain feedback from important in this e"ort by an Advisory Board. !e members of the in American schools geography is a part of the social constituencies for the reports, the leadership team Advisory Board have reviewed and endorsed the reports studies curriculum and is not recognized as including reached out to organizations in related #elds to help and are committed to helping the partners achieve their the substantial components of physical, life, and earth construct a formal Review Board for the reports. Eleven dissemination goals. !e Advisory Board members are sciences included in Geography for Life. organizations were contacted and asked to nominate listed on page 5. For readers who are unfamiliar with the contents of members or representatives of their organizations to Geography for Life, we encourage them to familiarize review each of the reports. Of these, eight organizations Scope and Terminology themselves with it prior to reading this report. We also nominated individuals. From this group, 15 individuals In a subject-speci#c educational project such as this, it is recommend that all readers bear in mind that when provided reviews of one or more reports each. !e important to be explicit about the scope of the project. the term geography appears in this report, it refers to nominating organizations and the members of the !is is particularly important for geographic education, the full range of knowledge, skills, and perspectives Review Board are listed on page 15. because there is so much confusion about the nature of described in Geography for Life 1, not only those that are Final Preparation geography and its relationship to the K–12 curriculum. taught under the label of “geography” in schools today. In particular, the range includes elements of the social Following the completion of the public and Review For this project, the partners chose to use the national sciences, which typically are taught as part of the social Board reviews, the committees carefully reviewed all standards document Geography for Life: National studies curriculum in American schools, and elements of the comments received. !e committee chairs, Geography Standards, Second Edition, as the scoping of the physical, life, and earth sciences, which typically co-chairs, and directors met in Washington, DC, in document (Heffron & Downs, 2012). Geography for are taught as part of the science curriculum. April 2012, to discuss the themes that arose from the Life lays out a scope for geographic education that cuts feedback and construct plans for how to address them. across the traditional boundaries of social studies and Acknowledgments Each of the committees then met in late April/early science in American schools, reflecting the fact that May to discuss the feedback in greater detail, work on geography is concerned with both the physical world !is material is based upon work supported by the their responses, and #nalize plans for completing the and the social world. National Science Foundation under Grant No. DRL- #nal drafts of the reports. 1049437. Any opinions, #ndings, and conclusions or While the scope of geographic education as defined recommendations expressed in this material are those of Final drafts of the reports were submitted for editing by Geography for Life is consistent with the way the author(s) and do not necessarily re%ect the views of and layout in August 2012. academic geographers define the field of geography, it the National Science Foundation. is inconsistent with the way the term geography is used Dissemination in most American schools and with the understanding Following the publication of the reports, the four of the term by many members of the general public.

partner organizations will engage in a dissemination In most American schools and in the minds of the 1 In the original proposal to NSF, the partners used the term e"ort in order to bring the reports to the attention of general public, the term geography refers to a set of basic geographical sciences to describe the project’s scope. For the sake of readability, we decided to use geography in the project reports. !is their target audiences and to educate policy makers, map-reading skills; a collection of facts about place change in terminology does not re%ect any shift in the focus of the funders, and front-line educators about the reports’ names and locations; and a body of information about initiative in the intervening time.

The Road Map Project | Assessment Report | Preface 13 of 75 Geography Executive Context and Purposes of Geography Example Appendices Preface Assessment Recommendations References Summary Goals Assessment Assessment Framework A, B, C Framework

!e Assessment Committee would like to acknowledge helped shape our thinking on geographic practices. !e throughout the report development process. We would the contributions of many people to this report. Two of Committee received very valuable feedback from the especially like to recognize Jill Wertheim, the Assess- the original Committee members, Roger Downs and members of the Review Board, members of the other ment Committee Research Director, who did a fantastic Sharif Shakrani, were unable to serve on the Com- Road Map Project committees, and participants in the job of simultaneously conducting an original research mittee through the entire project but made important public review. !e reports were improved immeasur- study, coordinating a committee distributed across the contributions during the periods when they were able ably from this feedback. In addition, L. Karina Nabors United States, and writing and editing this report. Last, to participate. Four talented individuals formed the assisted with data collection for the study on geography we would like to acknowledge the important role that expert panel that contributed to the example assessment assessments, and Lafayette Cruise helped with many as- the leadership team played in helping us to create this framework described in Chapter 5: Robert Austin (Utah pects of the report as an intern at National Geographic. Department of Education), James Dunn (University of report and in working together to make the Road Map Northern Colorado), Beth Ratway (American Institutes Finally, the chair and co-chair would like to acknowl- Project into a single, cohesive e"ort, rather than a set of for Research), and Sandra Schmidt (Columbia Teacher’s edge the insightful contributions of the entire Commit- independent committees. None of this would have been College). !e Committee is very grateful to the present- tee and express our gratitude for their patience, respon- possible without the capable leadership and coordina- ers at the Workshop on Geographic !inking, who siveness to requests on short notice, and their hard work tion of Virginia Pitts, the Road Map Project Director.

The Road Map Project | Assessment Report | Preface 14 of 75 Geography Executive Context and Purposes of Geography Example Appendices Preface Assessment Recommendations References Summary Goals Assessment Assessment Framework A, B, C Framework

Road Map for 21st Century Geography Education Project Review Board

!e following organizations nominated reviewers to serve on the Review Board of !e following individuals nominated by these organizations reviewed one or more the Road Map for 21st Century Geography Education Project: of the Road Map Project Committee reports:

American Association for the Advancement of Science (AAAS) Assessment Committee Report Instructional Materials and American Federation of Teachers (AFT) Ann Benbow (AGI) Professional Development Committee Report American Geosciences Institute (AGI) John Lee (NCSS) John All (AAAS) Council of State Social Studies Specialists (CS4) Glen MacDonald (AAAS) Stephanie Hartman (CS4) National Board for Professional Teaching Standards (NBPTS) Lauren Mitterman (NBPTS) John Lee (NCSS) National Council for the Social Studies (NCSS) Sheryl Mobley-Brown (AFT) Sheryl Mobley-Brown (AFT) National Education Association (NEA) Dean Nakanishi (NBPTS) Kevin O’Brien (NBPTS) North American Association for Environmental Education (NAAEE) Alan Reid (NAAEE) Judith Wilson (NEA)

Geography Education Research Committee Report Fay Gore (CS4) By participating in this review process, these organizations and individuals Jackie Huntoon (AGI) made an important contribution to the Road Map for 21st Century Geography John Lee (NCSS) Education Project. However, they were not asked to endorse the reports that they reviewed, so the participation of these organizations and individuals does Glen MacDonald (AAAS) not constitute an endorsement of the reports. While the members of the Review Sheryl Mobley-Brown (AFT) Board were nominated by organizations, they did not represent the views of Bora Simmons (NAAEE) their organizations in the review process. Robin Wheeler (NEA)

The Road Map Project | Assessment Report | Preface 15 of 75 Geography Executive Context and Purposes of Geography Example Appendices Preface Assessment Recommendations References Summary Goals Assessment Assessment Framework A, B, C Framework

Road Map for 21st Century Geography Education Project Presenters

Workshop on Geographic Thinking

Washington, DC, June 16–17, 2011

!e following invited speakers presented at a Richard Duschl Janice Monk workshop on geographic thinking convened by all !e Pennsylvania State University University of Arizona three committees of the Road Map for 21st Century Carol Gersmehl Daniel Montello Geography Education Project in June 2011: New York Geographic Alliance and University of California, Santa Barbara Thomas Baerwald Renaissance Charter School Alec Murphy National Science Foundation Phil Gersmehl University of Oregon Douglas Batson Michigan Geographic Alliance and Nora Newcombe National Geospatial-Intelligence Agency New York Center for Geographic Learning Temple University Scott Bell Patricia Gober Jeanette Rice University of Saskatchewan Arizona State University Rice Consulting, LLC Sarah Brinegar Susan Hanson Peter Seixas U.S. Department of Justice Clark University University of British Columbia Roger Downs Kim Kastens !e Pennsylvania State University Columbia University

Lynn Liben The Pennsylvania State University

The Road Map Project | Assessment Report | Preface 16 of 75 Geography Executive Context and Purposes of Geography Example Appendices Preface Assessment Recommendations References Summary Goals Assessment Assessment Framework A, B, C Framework

Chapter 1: Context and Goals for the Road Map for 21st Century Geography Education Project

The State of Geography Education Assessments of geographic concepts and skills confirm grade, more than 30% of students scored below “basic,” in the United States the failure of our educational system in geography, indicating that they had not mastered even foundational indicating that the overwhelming majority of American geographic concepts or skills. This report is one of three synthesis reports on geog- students are geographically illiterate. The 2010 National From the NAEP results and other data, we conclude raphy education from the Road Map for 21st Century Assessment of Educational Progress (NAEP), known Geography Education Project. The Road Map Project that an overwhelming majority of high school graduates as “The Nation’s Report Card,”(National Center for are not prepared to do the ordinary geographic reasoning has been a collaborative effort of four national orga- Education Statistics, 2011) found that fewer than 30% that is required of everyone in our society in the course nizations: the American Geographical Society (AGS), of American students were proficient in geography; of caring for themselves and for their families, making the Association of American Geographers (AAG), the more than 70% of students at fourth, eighth, and 12th consequential decisions in the workplace, and partici- National Council for Geographic Education (NCGE), grades were unable to perform at the level that is ex- pating in the democratic process. Furthermore, we and the National Geographic Society (NGS). These or- pected for their grade (NCES, 2011, Figure 1). At 12th conclude that more than 30% of high school students ganizations share a concern that the dismal state of K–12 geography education across the United States is a threat Figure 1. Comparison of Results for Students in Grades 4, 8, and 12 on National Assessment of to our country’s well-being, and by extension, the well- Educational Progress (NAEP) Geography Test in 1994, 2001, and 2010 being of the global community. The project partners Trends in NAEP Geography Achievement Results share the belief that geography education is essential for for 1994, 2001, and 2010 preparing the general population for careers, civic lives, and personal decision making in contemporary society. Grade 4Grade 8Grade 12 It also is essential for the preparation of specialists ca- 100% 100% 100% pable of addressing critical societal issues in the areas of social welfare, economic stability, environmental health, 80% 80% 80% and international relations. The Road Map Project part- 60% 60% 60% ners fear that by neglecting geography education today, we are placing the welfare of future generations at risk. 40% 40% 40% While inspiring examples of highly effective geography 20% 20% 20% education can be found in every part of the United States, the amount of geography instruction that the 0% 0% 0% overwhelming majority of students receive, the prepara- 1994* 2001* 20012010 1994* 2001* 2001 2010 1994* 2001* 2001 2010 tion of their teachers to teach geography, and the quality of their instructional materials are inadequate to prepare Below basic Basic Proficient Advanced students for the demands of the modern world. Test administrations in which accommodations were not permitted Source: NCES, 2011

The Road Map Project | Assessment Report | Chapter 1 | Context and Goals 17 of 75 Geography Executive Context and Purposes of Geography Example Appendices Preface Assessment Recommendations References Summary Goals Assessment Assessment Framework A, B, C Framework

are so far behind that it is unlikely they will ever reach In addition, we need to provide young people with and events around the world and put appropriate proficiency. To compare with textual literacy, this level the opportunity to develop the understanding and responses forward for decision-makers. We need of geographic illiteracy is analogous to having 70% of interest to pursue the geography-dependent careers people who are able to operate on the ground in ev- high school graduates unable to read a newspaper edito- that are critical to our national interests. The Geo- ery kind of foreign context and can read the cultural rial and identify the assumptions, evidence, and causal Literacy Coalition, a consortium of businesses including and physical landscape appropriately. connections in its argument. Google, CH2M HILL, Esri, and the U.S. Geospatial This Road Map Project is taking place against a backdrop Intelligence Foundation, had the following to say about in which many members of the global community are The Importance of the importance of geography education for our nation renewing their commitment to geography education. In Geography Education (National Geographic, 2011): Australia, a national curriculum is being introduced for K–12 geography education is critical preparation for [America’s] inattention to [geography education] the first time. In England, geography is a component civic life and careers in the 21st century. It also is stands in contrast to the demand for geographically of the recently introduced English Baccalaureate. In essential for postsecondary study in a wide range of literate individuals in the workforce. There is substan- most of the world, geography holds a higher place in the fields, from marketing and environmental science, to tial demand in both the public and private sectors for K–12 curriculum than it does in the United States. In international affairs and civil engineering. people who have the ability to interpret and analyze most countries, geography is required every year through geographic information. The number of jobs for such age 16, in addition to history or other social studies Everyone in modern society faces personal decisions analysts is growing rapidly, while the supply of Ameri- subjects. In fact, the United States is almost unique in that require geographic reasoning. These decisions, such cans who can fill them is not. By not preparing young its treatment of geography as part of a single curriculum as where to live and how to travel from place to place, people for careers that depend on geographic reason- with history, government, and economics. can have an enormous impact on one’s life. We also ing, we are leaving ourselves vulnerable. must make decisions that have far-reaching consequenc- In our global economy, the understanding and ana- The Road Map Project partners believe that we, as a es, such as which products to buy and how to dispose lytical skills developed through geography education society, have a responsibility to prepare all young people of them. While these decisions may seem insignificant, are essential to make well-reasoned decisions about for their personal needs and civic responsibilities, and when they are multiplied by the number of people where to conduct business, how to conduct business we have a further responsibility to prepare sufficient making them each day, they have enormous cultural, in particular locations, and how to transport materi- numbers of young people for geography-dependent economic, and environmental repercussions for other als and goods from one location to another. Critical careers. We are not currently living up to those people and places. Finally, in our democratic society, we business choices such as where to build facilities, responsibilities, and we fear the consequences that our all participate in societal decision making about public how to design a supply chain, and how to market to society will suffer if we continue to neglect geography health, social welfare, environmental protection, and different cultures all require geographic reasoning. education. international affairs. In this era of such global challenges as ethnic and religious conflict, growing populations These skills are equally important for emergency The Need for a “Road Map” for in poverty, increasing competition for limited natural preparedness, defense, intelligence, and diplomacy. Geography Education resources, and degradation of the environment, it is es- In our government and military, we need individuals sential that all members of society be prepared to make who understand the dynamics of specific locations Over the past several decades, a small but dedicated these decisions. Geography education helps prepare well enough to prepare for and respond to emergen- community of geographers and educators has harbored people for these tasks. cies. We need analysts who are able to track people concerns about the state of geography education and

The Road Map Project | Assessment Report | Chapter 1 | Context and Goals 18 of 75 Geography Executive Context and Purposes of Geography Example Appendices Preface Assessment Recommendations References Summary Goals Assessment Assessment Framework A, B, C Framework

has worked diligently to improve geography education. allocating resources to improve geography The Instructional Materials and Professional Their greatest success has been in establishing a firmer education that accurately reflect its importance Development Committee considered the place for geography in K–12 education. The Elementary for our society. current state of the instructional materials for and Secondary Education Act (ESEA) of 2001 (January teaching geography and the preservice and !is work targets the three audiences that are in the best 8, 2002) recognized geography as a core academic inservice education that teachers who are respon- position to e"ect improvement in our system of public sible for geography education receive. Based on subject, and all 50 states now have K–12 standards education: this analysis and a review of the literature on the for geography. Geography has been included in the design of instructional material and the design of 1. Front-line professionals: educators, teacher National Assessment of Educational Progress since teacher professional development, the Commit- educators, developers, and researchers who 1994, and the College Board established an Advanced tee formulated recommendations and guidelines directly influence instruction, assessment, Placement exam for Human Geography in 2001. for both instructional materials and professional and research; development that will lead to improvements in However, these successes in improving the place of 2. Policy makers: individuals at national, state, and instruction and in learning outcomes. geography in the educational system have not been fol- local levels who establish the goals and processes The Assessment Committee studied the current lowed up with the levels of effort or resources necessary for public education; and state of assessment in geography and reviewed to bring about widespread improvement in the quality its history. Based on their analysis of existing 3. Funders: decision-makers in government and of instruction. As a result, educators and students who assessment practices and a review of the literature private organizations who provide the funding to have had the good fortune of being impacted directly on assessment as a support for improving support public education. by the efforts of the geography education reform com- educational outcomes, the Committee formulated munity have benefited enormously, but they represent guidelines for developing assessment instruments In planning the project, the partners identi#ed #ve criti- a small minority. As measured by NAEP, there has been and for conducting assessment that will lead to cal issues for improving geography education: no broad improvement in students’ learning of geogra- improvements in instruction and outcomes. phy during the 17−year period of testing. 1. preparation and professional development The Geography Education Research Committee of teachers, reviewed the existing education and cognitive The project partners launched the Road Map Project 2. instructional materials to support classroom science research literature to identify gaps in with the goal of increasing the scale and accelerating instruction, our ability to answer significant questions about the pace of efforts to improve geography education geography education based on research. Drawing 3. assessment of learning outcomes and to meet our responsibility to prepare young people on this analysis, the Committee formulated for the world they will inherit. The partners have two instructional effectiveness, recommendations for research questions and goals for this work: 4. research on teaching and learning, and approaches that will build a knowledge base to guide improvement efforts for geography • first and foremost, to make future efforts to 5. cultivation and maintenance of public support. education in the future. improve geography education more strategic, focused, and coherent, so they can have greater !e partners divided these issues among four e"orts, For the #nal issue—developing and maintaining public and more enduring impact; and deciding to address the #rst four issues through support for geography education—the partners did not • second, to provide a rationale for establishing synthesis reports to be developed by three committees of believe the existing knowledge base on public beliefs requirements for geography education and experts identi#ed by the project partners: and attitudes about geography education would sup-

The Road Map Project | Assessment Report | Chapter 1 | Context and Goals 19 of 75 Geography Executive Context and Purposes of Geography Example Appendices Preface Assessment Recommendations References Summary Goals Assessment Assessment Framework A, B, C Framework

port the development of a synthesis report at this time. the goals for K–12 geography education should meet. 2010; National Research Council, 1997). As described Instead, the partners initiated a pilot study of public Specifically, goals for geography education should: in a recent National Research Council report (NRC, beliefs and attitudes under the direction of the Ameri- 1. reflect the essence of geography as defined by 2010), geography involves: can Geographical Society. geographers; documenting, analyzing, and explaining: 1) the lo- Establishing a Destination: Goals for 2. convey the qualities of geography that capture its cation, organization, and character of physical and K–12 Geography Education distinctive benefits as a subject of study; and human phenomena on the surface of Earth; and 2) 3. focus on the portions of geography that have the the interplay of arrangements and processes, near The value of a road map is that it enables you to select greatest value for students and society. and far, human and environmental, that shape the a route to your destination. Therefore, the first step in evolving character of places, regions, and ecosys- developing our Road Map for geography education was We approached the challenge of defining the goals for tems (p. 10). establishing a common destination. In education, des- geography education from two perspectives—those !is report characterizes geography as being forward- tinations are expressed in terms of learning outcomes, of geographers and educators. To explore the perspec- thinking and essential to society for key issues includ- so in the case of geography education, we will be able tive of geographers, we surveyed the existing literature ing sustainability, economic stability, national security, to say that we have reached our destination when our on the nature of geography, and we convened current schools make it possible for all students to achieve the thinkers and practitioners for a workshop on “geo- and response to environmental change. graphic thinking.” At this workshop, we invited a learning goals for geography that we have set for them. A consensus also has evolved in recent decades about wide variety of academic and practicing geographers, the key themes of geography. Pattison (1964) identi#ed Because the national geography standards were devel- cognitive scientists, and individuals with other relevant geography’s core as consisting of four “traditions,” the oped through an earlier collaboration of the project perspectives to present on what it means “to think like partners, they represent a logical choice of “destina- spatial tradition, the area studies tradition, the human- a geographer” or “to do geography.” To explore the tion.” However, the members of the Road Map Project land tradition, and earth science tradition. Taa"e perspective of geography educators, we examined the committees thought we should use this opportunity to (1974) identi#ed three key organizers for geography: history of efforts to conceptualize geography education consider alternatives as well. Therefore, as a collabora- spatial organization, area studies, and human-land during the past half century. We summarize the find- tive effort across all three committees, we conducted relationships. Contemporary geographers agree that the ings of these investigations below. an investigation into what it means to “do geography” discipline focuses on a similar set of core ideas: spatial- in the 21st century and what that implies for the goals Geographers on Geography ity, human-environment interaction, interconnections of K–12 geography education. The remainder of this between places, and place-based and regional analysis We started our investigation with a review of the ways chapter describes that process and its outcomes. (Abler, 1987; Baerwald, 2010). that geographers have defined geography in recent Establishing goals for geography education is no small decades. While there is great diversity of opinion Because geographers work on many of the same ques- challenge because geography is a broad field and it among geographers about where the boundaries of tions and problems as specialists in other #elds, they is constantly evolving. Fortunately, geographers and geography lie, there is considerable consensus about have faced the challenge of di"erentiating geography others have wrestled with this challenge for generations, its core. Geographers engage in a range of activities from those #elds. Susan Hanson confronted this chal- and we were able to benefit from that work. Our related to space, place, and the dynamic interactions of lenge in a presidential address to the Association of investigation was guided by three criteria that we believe agents within and across spaces and places (Baerwald, American Geographers. In this presentation, Hanson

The Road Map Project | Assessment Report | Chapter 1 | Context and Goals 20 of 75 Geography Executive Context and Purposes of Geography Example Appendices Preface Assessment Recommendations References Summary Goals Assessment Assessment Framework A, B, C Framework

(2004) described the unique aspects of geography as tion and at the in%uence they had. We summarize The unconventional HSGP units entered a challenging “the geographic advantage,” and she enumerated four what we learned in the paragraphs that follow. Across implementation environment in the late 1960s and early aspects of this advantage: these e"orts, we observed two important trends: (1) an 1970s. The objective was to create a dynamic, participa- 1. Geography considers the relationships between increase over time in their richness and clarity, and (2) tory learning environment in which students observed humans and environments. Because of the an ongoing struggle to present a balance between what that geography is a conceptually rich and useful subject traditional separation of social and physical it means to “understand” geography and what it means for daily life in their communities and the larger world. sciences, other disciplines tend to focus on one to “do” geography. Although the units were favorably reviewed and support- or the other. The High School Geography Project (1963 to 1971). ed with teacher training, they differed significantly from 2. Geography recognizes the importance of spatial Today’s efforts to improve geography education have their existing materials and teaching practices. Further, the variability. Geography offers unique methodolo- roots in the wave of educational reform initiatives that learning outcomes that the inquiry-based units targeted gies for investigating the way phenomena vary could not be assessed using conventional testing. Conse- with location and explaining the place-dependen- followed the Soviet Union’s launch of the Sputnik satel- quently, the HSGP was not widely adopted in American cy of processes. lite in 1957. One of these initiatives targeted geography education, and it set a tone that has influenced all subse- high schools. However, the project did engage a com- 3. Geography considers the multiple and inter- quent geography education reform efforts. The NSF- munity of academic geographers in K–12 education for locking geographic scales at which processes the first time in more than a decade, and it introduced a operate. Geography also offers unique techniques funded High School Geography Project (HSGP) was an for studying phenomena and how they play out instructional materials development initiative with the concept of the goals and methods of geography educa- over multiple spatial scales. goal of transforming high school geography (Association tion to a new generation of educators. These two impacts helped to lay the groundwork for the next wave of reform 4. Geography integrates spatial and temporal analy- of American Geographers, 1966). In a reflection on the sis. With its focus on spatial variability, geogra- project, the project director said, “With little hesitation, efforts in the early 1980s. phy offers unique techniques for integrating the teachers [who were consulted in the design of the HSGP] The Guidelines for Geographic Education (1984). analysis of variation over time with analysis of voiced the same litany of problems…dull textbooks, The next influential effort to reconceptualize geography variation over space. Many other disciplines have inadequately trained teachers, simple factual content… education began in the early 1980s following the pub- focused on analysis of temporal variability with- training in history not geography, lack of emphasis on lication of A Nation at Risk: The Imperative for Educa- out attention to the spatial dimension. geography in schools of education...” (Helburn, 1998, tional Reform (National Commission on Excellence in p. 212). HSGP attempted to address many of these Evolving Conceptions of the Goals of Education, 1983), which, like the launch of Sputnik, Geography Education concerns by creating instructional materials that engaged triggered a wave of educational reform efforts across the students and teachers in asking and answering geographic In addition to looking at how geographers have charac- curriculum. In 1984, a joint committee of the Associa- questions using data and simulations, and by building terized geography in recent decades, we also looked at tion of American Geographers and the National Coun- professional development opportunities around the cur- the goals that geographers and educators have articu- cil for Geographic Education published the Guidelines ricula. Essentially, HSGP was an attempt to reconceptual- lated for geography education over that same period. for Geographic Education, which was designed to provide ize geography education as the integration of geography a clear, comprehensive set of national goals for K–12 During the past 50 years, four e"orts to conceptualize inquiry and geographic understanding. the goals of geography education have had nationwide geography education (Joint Committee on Geographic in%uence. In our investigations, we looked both at the In practice, the long-term impact of HSGP turned out Education, 1984). The Guidelines established a concise ways they characterized the goals of geography educa- to be more a result of its ideas than its implementation. framework for geography teaching that would be widely

The Road Map Project | Assessment Report | Chapter 1 | Context and Goals 21 of 75 Geography Executive Context and Purposes of Geography Example Appendices Preface Assessment Recommendations References Summary Goals Assessment Assessment Framework A, B, C Framework

adopted in schools, in teacher preparation programs, These seminal publications extended the teaching ex- The Guidelines, which had a much broader impact than and among publishers of geography texts and curricu- amples in the Guidelines, and they were widely distrib- the HSGP, led to a broad-based reconceptualization lum materials. The Guidelines described geography as uted, increasing the influence of the Guidelines. of the content of geography in mainstream education. consisting of three basic elements: However, its influence was largely limited to the The impact of the Guidelines was impressive. The 1. a geographic perspective (spatial and ecological conception of content in terms of the five themes it ways of viewing the world); publication was remarkably successful in achieving presented. The Guidelines’ depiction of geography as an widespread awareness of the five fundamental themes. integration of content, perspectives, and skills was largely 2. fundamental themes (Location, Place, Human Educators and curriculum developers found the five Environment Interaction, Movement, and Region); overlooked. themes to be memorable, relatively easy to understand, and Geography for Life: National Geography Standards and easy to apply in teaching geography. Thus, the 3. core skills (asking geographic questions, (1994). The next major effort to articulate the goals themes were widely integrated into school curriculum acquiring geographic information, presenting of geography education began in response to federal geographic information, analyzing geographic guidelines, preservice and inservice professional devel- legislation enacted in 1989. The Goals 2000: Educate information, and developing and testing opment, and instructional materials produced by pub- America Act (1994) was passed in response to a renewed geographic generalizations). lishers, school districts, and professional organizations concern about the state of education in the United through the concerted efforts of the nascent Geography States. As a result of concerted efforts by the geography With these three elements, the Guidelines continued the Alliance network sponsored by the National Geographic education community, geography was included as one effort begun with the HSGP to present a vision of geog- Society. To this day, the five themes continue to influ- of the five core subjects in the America 2000 reform raphy that integrates knowing with being able to do. ence geography education in many school settings and plan. This recognition resulted in funding to create a Following the publication of the Guidelines, the teacher preparation programs. national standards document for geography. (It was in Association of American Geographers, the American this era that the term “standards” was introduced into Unlike the content themes, however, the geographic Geographical Society, the National Council for the educational policy lexicon.) Geographic Education, and the National Geographic perspectives and skills in the Guidelines received scant With funding from the U.S. Department of Education, Society joined together to create the Geography attention. They were largely overlooked in subsequent the National Endowment for Humanities, and the Education National Implementation Project (GENIP), materials development and professional development National Geographic Society, the four GENIP partners which aimed to translate the Guidelines into practice. efforts. While the five themes were consistent with the launched a standards-writing project. Over two years During the ensuing five years, GENIP produced two general focus on knowledge of the educational reform additional documents to help educators to implement efforts of the 1980s, the perspectives and skills in the with extensive feedback and advice from a broad range of reviewers, advisory groups, and testimony at numerous the Guidelines: Guidelines were not. Like the inquiry-based elements of the HSGP, integrating these perspectives and skills public hearings, a diverse group of scholars and teachers • K–6 Geography: Themes, Key Ideas and Learning created the first set of national standards for geography. into educational practices would have required a larger Opportunities (Geography Education National In 1994, the product of this effort was published: Implementation Project, 1987), and change than most educators were comfortable making, Geography for Life: National Geography Standards particularly because the reform efforts of the 1970s were • Geography in Grades 7–12: Themes, Key Ideas and (Geography Education Standards Project, 1994). Learning Opportunities (Geography Education widely criticized at that time for an excessive focus on National Implementation Project, 1989). “process” at the expense of “content.”

The Road Map Project | Assessment Report | Chapter 1 | Context and Goals 22 of 75 Geography Executive Context and Purposes of Geography Example Appendices Preface Assessment Recommendations References Summary Goals Assessment Assessment Framework A, B, C Framework

In contrast to the 26-page Guidelines, the 1994 edition the authors incorporated the application of geographic by educators and curriculum developers as were the five of Geography for Life was 272 pages long. Geography understanding into these content standards in two themes. Even today, many textbooks and professional for Life incorporated everything in the Guidelines in ways. First, two of the essential elements—The World in development programs still use the five themes as a cen- some form. For example, Geography for Life retained the Spatial Terms and The Uses of Geography—describe the tral organizing scheme. On the other hand, Geography for Guidelines’ three-part structure of perspectives, skills, application of knowledge and understanding as content. Life has had an impact on educational policy that exceeds and content. However, much was modified and added: For example, The World in Spatial Terms includes any other geography education document in the past 50

• The two geographic perspectives highlighted in using maps and other geographic representations and years. The release of Geography for Life provided impetus the Guidelines were maintained in Geography technologies to report information from a spatial for all 50 states and the District of Columbia to establish for Life: spatial and ecological. They also were perspective; using mental maps to organize information state standards for geography, and it provided a model for described in significantly greater detail than they about people, places, and environments in a spatial them to follow. Geography for Life’s content and structure had been in the Guidelines. context; and analyzing the spatial organization of were studied by the standards writers in every state, and • The skills identified in Geography for Life are people, places, and environments on Earth’s surface. its influence can be seen in nearly all of them. an elaboration of the skills described in the The Uses of Geography element describes the application Guidelines for Geographic Education. They of geography to interpret the past and ways to apply As in previous documents, the balance between perspec- are: asking geographic questions, acquiring geography to interpret the present and plan for the tives, skills, and knowledge that the authors of Geogra- geographic information, organizing geographic future. Second, for each content standard, the authors phy for Life presented was not as influential as desired. information, analyzing geographic information, described what students should be able to do with that Despite their prominence in Geography for Life, perspec- and answering geographic questions. standard’s content knowledge, implicitly reinforcing the tives and skills are not nearly as well-represented in state • Instead of the five themes discussed in the importance of applying geographic knowledge. standards as the content standards presented in Guidelines, Geography for Life organized content the publication. around six essential elements (The World in Finally, Geography for Life helped to provide a well- Spatial Terms, Places and Regions, Physical Geography for Life: National Geography Standards, rounded picture of modern geography by providing Systems, Human Systems, Environment and Second Edition (2012). In 2007, the members of discussions of the nature of geographic inquiry and Society, and The Uses of Geography). These GENIP decided it was necessary to revise the national essential elements were, in turn, made up of 18 discussing why the study of geography is important. geography standards to reflect changes in the discipline content standards. Geography for Life offered existential, ethical, intellectual, of geography and in the world. The second edition and practical reasons why individuals should learn While Geography for Life took a large step toward of Geography for Life: National Geography Standards geography, and the publication described how society presenting a picture of geography as integrating (Heffron & Downs, 2012) maintained the spatial and benefits from having geographically informed citizens. knowing and doing through its elaborate description of ecological perspectives and the 18 content standards perspectives and skills, the authors were restricted by the Like the Guidelines for Geographic Education a decade of geography, and it extended and elaborated on the constraints imposed on national standards documents earlier, Geography for Life had a broad national impact geographic skills section. Reflecting an important at the time. Specifically, they were permitted only to on mainstream education. However Geography for Life’s change in the world since 1994, it incorporates geospa- use the term “standard” to label content objectives. impact on classroom practice was largely indirect. Its tial technologies for problem-solving into many of the For that reason, neither perspectives nor skills were direct impact was on educational policy. The publica- standards. The writing team also completely revised the described as standards in Geography for Life. However, tion’s six essential elements were not as widely taken up concepts and performance expectations throughout the

The Road Map Project | Assessment Report | Chapter 1 | Context and Goals 23 of 75 Geography Executive Context and Purposes of Geography Example Appendices Preface Assessment Recommendations References Summary Goals Assessment Assessment Framework A, B, C Framework

content standards based, in part, on new research in the for Life captures the four components of the reality over the next decade by increasing the reach and learning and cognitive sciences. The new descriptions geographic advantage. effectiveness of efforts to improve geography education. use consistent language for cognitive activities drawn • Focus on the portions of geography that have Each of the committee goals is designed to address a from research in the learning sciences, and they reflect the greatest value for students and society: In its critical implementation issue: the preparation of teach- new understanding of developmental learning across the focus on the scientific aspects of geography with ers, the nature of instructional materials, the design and K–12 continuum. practical applications, Geography for Life focuses structure of assessments, and the research base to inform on the portion of geography that the committees educational decision making. However, the success of believe is most valuable for students to learn.2 The new edition continues to advance the notion that all of these efforts hinges on the ability of individuals While Geography for Life does not capture the geography education should be framed around core to communicate about the true nature of geography, full diversity or richness encompassed by modern ideas, many of which are applicable to peoples’ daily including the geographic advantage, to key stakeholders. geography, the committees think it captures the lives, as well as personal and community decision mak- For that reason, we extended our consideration of the ing and problem solving. This edition makes the case subset that will be most valuable for students’ personal, professional, and civic lives. goals of geography education beyond what they should that being an informed citizen requires knowing the be to how they should be expressed. In doing so, we iden- content of geography and being able to use geographic Describing the Destination: tified two important issues to address: (1) the need to reasoning and skills. Effectiveness and Balance present a view of the different aspects of geography that Choosing a Destination: Geography for Life is balanced and integrated; and (2) the need to clarify Across the history of efforts to reconceptualize geog- what it means to “do geography.” After careful review and consideration, all three raphy education summarized above, there has been an committees agreed that the second edition of Geography ongoing struggle to promote the multi-faceted nature of A Balanced and Integrated View for Life should serve as the “destination” for the Road geography as perspectives, skills, and content, which is of Geography Map Project, because it meets all three of the criteria we contrary to a tendency in the educational system to focus !e stereotypical view of geography as fact-based and had established for the goals of geography education: more narrowly on content. The multi-faceted view of descriptive has proven persistent, no doubt because • Reflect the essence of geography as defined geography presented by the second edition of Geography the stereotype corresponds to the experience of most by geographers: In its presentation of the for Life contrasts with the stereotypical view of geogra- American students and teachers for generations. In content standards, Geography for Life reflects phy as being about facts, in particular, the locations and practice, this “understanding gap” functions as a the central elements that geographers have names of places. While this stereotype could not be more source of resistance to any e"orts to change geography identified with geography. inaccurate as a description of the field of geography, it is education. Making a signi#cant change to geography • Convey the qualities of geography that capture distressingly accurate as a description of the geography education will require a change in the understanding of In its distinctive benefits as a subject of study: education that American students experience. geography by all stakeholders. Introducing new concepts its depiction of the perspectives and skills and its of subject matter has proven to be a di&cult challenge process-oriented content standards, Geography If it is successful, the Road Map Project will change this in the American educational system, but this is an

2 occasion where the geography education community We characterize the geography presented by the second edition of Geography for Life as scienti#c because it employs methods of inquiry and standards of evidence that are associated with contemporary scienti#c practice. !is subset of geography is sometimes referred to as the has the opportunity to learn from the experiences of geographical sciences. By referring to this geography as scienti#c, we are describing its methods, not its content. Geography for Life re%ects the consensus view of geographers that geography is concerned with both the social and physical worlds, and that it has a particular concern for other disciplines. For example, the backlashes that have the interactions between those worlds. confronted both math and science education reform

The Road Map Project | Assessment Report | Chapter 1 | Context and Goals 24 of 75 Geography Executive Context and Purposes of Geography Example Appendices Preface Assessment Recommendations References Summary Goals Assessment Assessment Framework A, B, C Framework

e"orts teach us how important it is to present reform as different facets of geography in education, rather than constitute geographic practice. In the course of our a process of integrating traditional and new approaches, teaching them separately. Educational research teaches research, we found an alternative in the science and rather than as a replacement of traditional with new. us that it is ineffective to separate learning of facts, mathematics education literature—the word practice has concepts, and reasoning because they need to be used been adopted in recent years as a term for these kinds of For that reason, it is essential that we present a balanced together in practice. However, a traditional view, and activities we were trying to capture. In that literature, the view of geography that recognizes the importance of one that would feel more comfortable to many stake- term practice is used to describe the behaviors that com- learning the place names, locations, and terminology holders, would be that factual understanding should be prise scientific inquiry and problem-solving. A scientific that have characterized geography education histori- taught first, followed by conceptual understanding, and practice is a goal-directed set of actions that contribute cally, along with understanding powerful geographic then reasoning skills. to a scientific inquiry or problem-solving process. Some concepts, and being able to reason geographically. We of the scientific practices identified in the National must be careful not to present the new conception as Therefore, it is essential that we present a view of Research Council’s recent Framework for K–12 Science being a rejection or abandonment of what has been geography education that integrates learning of facts, Education are asking questions, defining problems, valued traditionally, but rather as an enhancement that concepts, skills, and reasoning at all levels from K to 12. developing and using models, constructing explanations, establishes a better balance. This lesson applies not only Geographic Practices and engaging in argument from evidence (NRC, 2012, to stakeholders that have been untouched by earlier p. 49). Practices are shared across disciplines, but they In reviewing the history of geography education reform, reform efforts, but also to those who have invested typically are conducted in different ways across different we see that the aspect of geography that has been taken in those reforms. For example, educators who have disciplines (NRC, 2011). In this respect, discipline-spe- up the least in schools is the application of geography embraced the richer conception of content presented by cific practices encode the perspectives of the discipline. Geography for Life and its precursors should see a focus understanding to answer questions or to solve problems. on geographic reasoning as an enhancement to their Where the articulation of the five themes in the Working from the skills described in Geography for Life, efforts, rather than as a replacement of them. Guidelines led to a broader understanding of geography we identified six categories of geographic practice. Each content among the educators who were reached by it, of these categories represents an aspect of geographic in- To help stakeholders understand the value of this multi- historically there has been no comparable broadening quiry or problem-solving, and encompasses specific prac- faceted geography (and to motivate them to support it), in the understanding of the practices of geography. tices that, either independently or in combination, can it is essential that we communicate the limitations of achieve a reasoning goal (Table 1). More detailed descrip- As a result, all three committees have paid special at- the traditional focus of geography education on its own tions of the practices, along with examples representing tention in their work to the question of how to ensure and the value of the additional components for learners. how they are used by practicing geographers, ordinary that “thinking geographically” and “doing geography” It is essential that we do so in terms that are meaningful people, and classroom instructors, can be found through- become integrated into classroom practices in the next to stakeholders (e.g., “college and career readiness” is out the three Road Map Project committee reports. the discourse of educational policy as this report is generation of geography education reform. Over the being prepared, as well as preparation for personal and course of our work, we identified terminology as an Because it suited their goals better, the Geography civic life). issue. Geography for Life uses the term skills to describe Education Research Committee condensed these six the activities that constitute the doing of geography. categories into a smaller set. !e Committee combined For pedagogical purposes, it also is important that However, concerns were raised by how well the term acquiring, organizing, and analyzing geographic we communicate the importance of integrating the skills describe the complex, goal-directed behaviors that information into a single category, and also combined

The Road Map Project | Assessment Report | Chapter 1 | Context and Goals 25 of 75 Geography Executive Context and Purposes of Geography Example Appendices Preface Assessment Recommendations References Summary Goals Assessment Assessment Framework A, B, C Framework

answering questions and designing solutions with Table 1: Geographic Practices 3 communicating geographic information. !us, the Committee’s three categories are: Categories Practices

1. Formulating geographic questions; a. Identify problems or questions that can be addressed using geo- Posing geographic graphic principles, models, and data; express problems and questions 2. Acquiring, organizing, and analyzing geographic questions in geographic terms. information; and

3. Explaining and communicating geographic a. Identify geographic data that can help to answer a question or solve patterns and processes. a problem. Acquiring geographic Mapping a Bright Future information b. Collect data (including observations and measurements) about geo- graphic phenomena, and/or gather existing data to help answer In this chapter, we have presented an overview of the a question or solve a problem. rationale and goals for the Road Map for 21st Century Organizing geographic a. Organize data and create representations of data to help solve a prob- Geography Education Project. The project is motivated information lem or answer a question. by a concern for the current state of geography education and the slow progress partners and others have made in a. Identify data analysis strategies that can be used to help solve a prob- improving it. By identifying promising strategies in key lem or answer a question. areas, we aim to mobilize and focus resources in ways that will increase the magnitude and pace of improvement. Analyzing geographic b. Find and describe spatial and temporal patterns in data, or find data information that matches a pattern, to help solve a problem or answer a question. The remaining chapters in this report provide an

analysis of key issues for geography education, and offer c. Construct an explanation or prediction for phenomena by comparing recommendations for how to focus improvement efforts data to a model or theory. during the coming decade. In doing so, this report joins the other Road Map Project reports in laying out a path a. Construct an answer to a question or a solution to a problem using geographic principles, models, and data. toward the destination described in Geography for Life— Answering questions an integrated geography education that balances learning and designing solutions b. Evaluate one or more answers to a question or solutions to a problem of knowledge, understanding, and practices. using geographic principles, models, and data.

Communicating a. Inform or persuade an audience using geographic principles, models, geographic information and data.

3 While the categories and practices are listed sequentially in the table following a widely used model of inquiry and problem-solving, we make no assumption that they will or should be conducted in that order in practice.

The Road Map Project | Assessment Report | Chapter 1 | Context and Goals 26 of 75 Geography Executive Context and Purposes of Geography Example Appendices Preface Assessment Recommendations References Summary Goals Assessment Assessment Framework A, B, C Framework

A Road Map for Geography support the development of high-quality assessments geography education outlined in Chapter 1. We also Assessments that advance the goals of education reform. describe how this general framework can be used as a blueprint for specific assessment frameworks targeted In Chapter 3, we consider the current state of assess- This particular report focuses on assessment and how at specific learning objectives and learner populations. assessment can be used as a mechanism for improving ment in K–12 geography education. We begin with a geography education. The Leadership Team of the Road discussion and comparison of existing frameworks for In Chapter 5 we describe how AFGS21 is designed to Map for 21st Century Geography Education Project large-scale assessments in geography and closely related be used by showing an example of a specific assessment identified assessment as a topic for a consensus report fields. We then report on a study we conducted of exist- framework. We describe the process of creating this because of the central role that assessment plays in all ing geography assessment items designed for classroom framework from AFGS21 for an end of unit exam for aspects of education reform, and because of the need and for large-scale assessments. We analyze how well a high school course. Chapter 3 also provides a detailed for a critical review of how assessments are being used current assessment practices meet the goals outlined in description of the steps taken to create this framework, to monitor and improve geography education. In this the standards and frameworks, and how well they satisfy and how to use the framework to develop assessments report, we examine the roles that assessments can play the criteria for effective assessments. Based on this that meet the criteria for effective assessments laid out in advancing the goals of geography education and how analysis, we argue that there are weaknesses in current in Chapter 2. existing frameworks and assessments are currently serv- assessment practices that need to be addressed as part ing those roles. We present a framework for geography of any effort to improve geographic education, and that Finally, Chapter 6 presents recommendations for assessment that will contribute to the larger efforts to there is a clear opportunity for assessment to more ef- specific steps toward the implementation of the new improve geography education over the next decade. fectively support improvement of geographic education. approach to assessment. This approach, combined with the recommendations in the other two Road Map We begin our discussion in Chapter 2 with a descrip- In Chapter 4, we respond to the issues raised in the Project reports, will lead to meaningful improvements tion of the various roles that assessment can play in the preceding chapters with a proposal for a new approach in the effectiveness of geographic education over the improvement of education. In order for assessments to to assessment in geographic education. We present this next decade. have the beneficial impact on educational practices and approach in the form of a new framework for the design outcomes, they must be thoughtfully designed, imple- of assessments that we call the 21st Century Assessment In Appendix A, readers can find additional details about mented, scored, and interpreted. Chapter 2 describes Framework for the Geographical Sciences (AFGS21), the study of existing assessments described in Chapter 3. some of the essential considerations for designing effec- and we describe how this framework builds upon the Appendix B has the full assessment framework described tive assessments, and how assessment frameworks can existing frameworks to move toward the goals for in Chapter 5.

The Road Map Project | Assessment Report | Chapter 1 | Context and Goals 27 of 75 Geography Executive Context and Purposes of Geography Example Appendices Preface Assessment Recommendations References Summary Goals Assessment Assessment Framework A, B, C Framework

Chapter 2: Improving Teaching and Learning Through Assessment

Assessments have always played an important role in our recommendations about how assessments should Di"erent types of assessments provide di"erent kinds education because of the information they can provide be used to improve teaching and learning in geography. of evidence. Formative assessments are used to for decision making. Teachers use information from as- We continue with a discussion about the challenges of monitor students’ progress during instruction to inform sessments to adjust their instruction. Schools and school designing assessments to serve these purposes, which the teacher of how to adjust instruction to suit their systems use information from assessments to place informs our recommendations about how to design as- needs. Formative assessments can be used to identify students, evaluate programs, and allocate resources. As- sessments to serve our recommended uses. gaps between a student’s current level of mastery and sessments play an important role in communication as the objectives for that student, and those gaps can be well. At the classroom level, the form and content of an Uses of Assessment Information used for selecting targeted instructional approaches. assessment informs students about what is expected of in Education Summative assessments are used following instruction them. At the system level, the form and content of as- to evaluate the outcomes of the instruction. !ey can Assessment is commonly viewed as a separate activity sessments serve to inform teachers, students, and other be used to measure students’ learning achievements from instruction. Indeed, in the Committee’s survey of stakeholders about the priorities of the system. geography assessments, fewer than 60% of geography and to inform programmatic changes to improve future instructional approaches. Because of these multiple roles, assessment is increasing- curriculum units included assessments (Appendix A). ly being recognized as a point of intervention for reform !e result of this separation is a missed opportunity Both formative and summative assessments provide of teaching and learning. For example, at the classroom for educators, students, parents, and policy makers to valuable information for decision making. In this level, assessments that reveal more information about improve teaching and learning (Black & Wiliam, 1998, report, we focus on four primary uses of information 2004a, b). By contrast, we believe that assessment must what students think, know, and can do enable that from assessments: teacher to make better decisions about instruction. At be integrally connected with teaching and learning 1. instructional decision making, the school and system levels, e"orts to change the focus (Figure 3). Each may move to the forefront at di"erent of instruction can be supported by changes in the focus points in time, but they must maintain a strong link 2. measuring individual achievement, of assessment. to the others. In this report, we focus primarily on the 3. evaluating programs, and assessment vertex of the teaching-learning-assessment 4. educational research. !e premise of this report is that assessment can be triangle, where evidence from assessment informs and used in these ways to improve teaching and learning improves teaching and learning. Assessments That Inform Instructional in geography. In a nutshell, the question that the Road Decision Making Figure 3. Teaching-Learning-Assessment Triangle Map Project Assessment Committee has investigated is Formative assessment is designed to inform ongoing how can assessment in geography support improvements in TEACHING LEARNING teaching and learning. Formative assessments can take teaching and learning of geography? many forms, from formal tests to oral questions posed In this chapter, we begin our consideration of that ques- on-the-%y, but the key feature of these questions is that tion with a discussion of the di"erent ways assessment they are intended to make students’ thinking apparent, data can be used in education. !is discussion informs ASSESSMENT so the teacher can monitor learning while instruction is

The Road Map Project | Assessment Report | Chapter 2 | Improving Teaching and Learning Through Assessment 28 of 75 Geography Executive Context and Purposes of Geography Example Appendices Preface Assessment Recommendations References Summary Goals Assessment Assessment Framework A, B, C Framework

still taking place. Because formative assessments moni- distract from other learning activities. !e introduction Assessment for Measuring tor students while a lesson or unit is under way, educa- of rapid assessment tools, such as classroom “clickers,” Individual Achievement tors can use the results from the assessment to adjust that allow a teacher to collect immediate information A second purpose for assessment is to measure individ- instruction to either bring attention to the students’ about students’ ideas with minimal disruption of the ual achievement. !is is done through assessments that particular needs or to move past objectives that students class, and computer-based assessments embedded follow instruction. !ese summative assessments typi- have already mastered. Students can use the results of in instructional activities that provide instantaneous cally are used to measure students’ competencies against formative assessments to understand where they have feedback to students and point them toward useful the benchmark learning goals; the results are used to made progress and where they need additional work. lessons, represent promising innovations for integrating determine if students su&ciently met those goals and E"ective instructional practices invariably involve ongo- assessments more seamlessly into learning. satis#ed their requirements, or the results are used to ing, informal formative assessment in real time every rank students relative to each other. Formative assessments, even informal ones, must have day (Wiggins & McTighe, 2005); nevertheless, forma- clear goals for the information that the educator is tive assessments may involve more formal assessment Assessments of achievement have more stringent trying to obtain. For these assessments to provide the tools, which can be used at critical points in an instruc- requirements for technical quality (i.e., validity and necessary information to guide teaching and learning, tional sequence. reliability) compared with formative assessments. !ese they must enable the educator to identify what the assessments typically take the form of tests that cover a Naturally, implementation of the information gathered students know, what they can do, and what they still broad span of learning objectives, which contrasts with from formative assessment is critical. If the information have to learn. !is requires that the assessment be based the targeted questions on a speci#c learning objective is not used, the purpose of formative assessment is on a clearly de#ned, #ne-grained target competency. typical of formative assessments. To characterize not achieved. If feedback is provided to individuals or !at is, formative assessment has a narrower focus when students’ competency across a domain, assessments groups of students, it is important to make sure that it compared with summative assessments. Each item of achievement tend to rely on extrapolation from focuses on strategies that students can use to advance focuses on a narrow scope of knowledge, practices, or a student’s performances based on a relatively small their learning. Feedback that focuses only on correctness competencies. Once a target competency, or construct, sample of the domain. If such a sample is to accurately often does not provide enough information to students has been described explicitly, e"ective assessment represent a student’s competency, the assessments about how to improve. For formative assessment to have strategies can be developed to identify where students should be held to stringent requirements for technical the greatest impact on improving learning, students are with respect to the target. quality. For example, they must be designed to ensure should be given the opportunity to use the feedback, that students interpret the items in the same way, and !e central bene#t of formative assessment for and teachers should verify that it was used appropriately. that their answers can be interpreted and evaluated improving teaching and learning is that it provides Experts generally discourage the use of formal scoring accurately and consistently. Enforcing high standards information to teachers and students that can help tailor or grading in formative assessment to maintain for technical quality ensures that decisions are made the teaching and learning process to the speci#c needs the focus on the learning process, rather than on based on valid, reliable, and fair information. of the students. Whether these are questions to students achievement (Black & Wiliam, 1998). in the course of a hands-on activity or quizzes at critical Assessments of achievement also have bene#ts for Because formative assessments are used during junctures during instruction, they can provide critical improving teaching and learning. However, rather than instruction, consideration must be given to the amount information about the progress of students and how to supporting short-term decision making within a unit of of time they require and the degree to which they focus the remaining instructional time. instruction as formative assessments do, they support

The Road Map Project | Assessment Report | Chapter 2 | Improving Teaching and Learning Through Assessment 29 of 75 Geography Executive Context and Purposes of Geography Example Appendices Preface Assessment Recommendations References Summary Goals Assessment Assessment Framework A, B, C Framework

longer-term decisions about students’ educational paths, time period than do formative assessments. !ey also have been communicated, they must still be interpreted such as whether they are prepared to move to higher enable improvement over a broader scale. !ey enable and applied to speci#c cases before they can improve levels, which courses and programs of study they should educators, administrators, and policy makers to make de- the quality of teaching or learning. So, while research enroll in, and whether they should be admitted to cisions about instruction based on one group of students #ndings have the bene#t of being general and of being selective educational programs or schools. that will impact subsequent groups of students. supported by high standards of evidence, they have the Assessment for Program Evaluation Assessment for Educational Research longest delay between their administration and their impact on teaching and learning. Another use for summative assessments is program Scholarly or scienti#c inquiry in geography education of- evaluation. Assessment information can be used to ten uses assessments to improve scienti#c understanding Considerations for the Design determine the e"ectiveness of materials, approaches, of teaching and learning. Assessments for research can of Assessments and educators. In the case of program evaluation, provide insight into what students think, how they learn, assessment information can be used to determine if and the e&cacy of di"erent approaches to instruction. !e design of sound assessments is a considerable chal- some aspect of instruction is achieving its goals or if lenge. In this section, we discuss important considerations As with the other types of assessments, assessments for it should be modi#ed in the future. For example, a for the design of assessments that can be relied upon to research must be reliable and valid for their intended uses, school district might decide to revise or replace a set of serve their purposes and improve teaching and learning. instructional materials based on summative assessment and they also require careful attention to how students data. Alternatively, an administrator might identify a interact with the assessment itself. Using assessments that Assessment development should always start with group of teachers to participate in targeted professional do not measure the appropriate outcomes, because of the following: poor technical quality or they are weakly aligned to the development because assessments reveal that their 1. determining who the user of the assessment students are struggling with the same set of skills. desired competencies, may lead to inaccurate conclusions. information will be (e.g., student, teacher, school !erefore, for data from assessments to be useful for in- and district administrator, state department As with the other purposes for assessment, assessments forming geographic research and development practices, of education); for program evaluation must be designed to serve technical rigor is of utmost importance. 2. identifying the population to be assessed the speci#c decision-making needs. Assessments for (e.g., grade level, special needs, geographic program evaluation must be sensitive to program e"ects, Assessments for research contribute crucial informa- distribution); so they can collect accurate information about how well tion for improving teaching and learning. However, the program is achieving those outcomes and identify they generally have the longest time-delay between 3. describing what information the user wants to obtain from the assessment (e.g., mental models, areas that require improvement. As in assessments assessment and impact on students. Whereas classroom subject mastery, achievement); and of individual achievement, assessments for program formative and summative assessments provide immedi- evaluation must meet high standards for technical ate information about speci#c students and program 4. clarifying how the information will be used quality, providing information that is valid, reliable, evaluation provides information about speci#c programs (e.g., teaching, course placement). and fair for making well-founded decisions. or educators, research provides information about students or educational approaches in general. Research Assessment tasks must be carefully designed to elicit the Assessments for program evaluation also have great value #ndings require su&cient quantities and diversity of content knowledge, practices, and cognitive processes for improving teaching and learning. Like assessments of data to support generalizations, which can require mul- that decision makers want to know about and that will achievement, they enable decision making over a longer tiple cycles of assessment. Once the research #ndings provide the appropriate data.

The Road Map Project | Assessment Report | Chapter 2 | Improving Teaching and Learning Through Assessment 30 of 75 Geography Executive Context and Purposes of Geography Example Appendices Preface Assessment Recommendations References Summary Goals Assessment Assessment Framework A, B, C Framework

Table 2. Characteristics of Assessment Design for Different Purposes Description Context of Administration Common Formats Practical Considerations

• Informs teaching Classroom assessment Informal and formal formative • Frequency of assessments • Informs learning assessments embedded in instruction, • Evaluating performance and feed- such as: ing back • Oral questioning • Sampling practices (e.g., one stu- • Observing dent, some students, whole class) • Informal evaluation (e.g., “stoplight- • Number of critical junctures or ing,” thumbs up/down) transition points in a unit

Decision Making • Constructed-response items • Time to administer and score an Inform Instructional Instructional Inform assessment • Selected-response questions

• Determines qualifications Classroom or large-scale assessment • Selected-response items • Sampling of content and cognition for advancement (includes classroom, school district, • Constructed-response items to be measured state, national or international level) • Student accountability • Performance tasks • Length of the assessments • Diagnose students’ problems • Interactive computer tasks • Time to administer the assessment Measure Measure • Logistics and delivery Achievement • Availability of technology

• Monitoring for program Large-scale assessment (includes • Selected-response items • Content and cognition to be improvement program, school or classroom reform, • Constructed-response items measured school district, state or national level) • Make comparisons between • Hands-on performance tasks • Length of the assessment programs • Interactive computer tasks • Time to administer the assessment • Measure program effectiveness • Interviews • Logistics and delivery • Observations • Availability of technology

Evaluate Programs Evaluate • Surveys

Provides empirical basis for model Small scale studies Different types of assessment can be Depending on the type of study, building/testing or for policy or practice used depending of the research being sampling issues should be considered: • Case studies decisions about, for instance: pursued, such as: • Ethnographies • Representative of high and low • curriculum/program development, • Interviews achievement • Design studies • teacher training • Observations • Representative of different demo- • Quasi-experiments • assessment • Constructed and selected response graphic groups • Randomized experiments • instructional program development items • Random sample • Observational (longitudinal) • Surveys • Matched samples Research Large-scale studies • Randomization to conditions • Quasi-experiments • Selectivity bias • Randomized experiments • Observational (longitudinal and cross-sectional)

The Road Map Project | Assessment Report | Chapter 2 | Improving Teaching and Learning Through Assessment 31 of 75 Geography Executive Context and Purposes of Geography Example Appendices Preface Assessment Recommendations References Summary Goals Assessment Assessment Framework A, B, C Framework

Pellegrino and colleagues developed a very useful three- !e interrelationship among the cognitive model, of the response. (!e combination of task and response component model for thinking about the design of as- observations, and interpretation shapes what (and how) is what we refer to in this report as an “item.”) !e sessments (National Research Council, 2001). !e three information can be gleaned from an item. Together, they speci#cation of these three components for a particular components of the model are cognition, observation, and form the conceptual basis underlying an assessment. De- assessment determines the format of the assessment. interpretation: cisions about the structural characteristics of assessment !e distinction between the components is important items, however, should be based on the user’s purpose or because, for example, a task that requires an extended • Cognition. For responses to an assessment to conducting the assessment. constructed response, such as an open-ended question, be considered representative of what students is not necessarily an assessment, but it can become one know, each assessment item must be grounded in As discussed above, assessments can serve four main if it is evaluated using a scoring system, even if it is an a model of how people learn that can be used to purposes: to assist teaching and learning, to measure informal one. On the other hand, the task would not interpret what students think and know about the individual achievement, to evaluate programs, and to domain based on their responses. be considered an assessment if the other components conduct research. One type of assessment cannot #t all are not employed; this is because there is no system • Observation. The assessment provides evidence purposes (NRC, 2001). !erefore, some considerations established for evaluating responses. !at is, an assess- for what the student can do (and by inference, are needed in determining the characteristics of an ment is incompletely speci#ed if it has only a task and what the student thinks—see below) while engag- assessment to #t its purpose. In Table 2, we outline the response, because there must be interplay between the ing in the specific assessment task. common types of assessments used for each of the four design of the response format and how the response will • Interpretation. Translating the evidence from purposes, how and when they tend to be used, and be interpreted. an assessment into a well-founded depiction practical considerations for their design. We also provide 1. Task type. Assessment tasks vary from asking of what the student knows about the domain a brief summary of the main characteristics, goals, and for a definition, to solving a problem, to requires more information than the assessment considerations for each assessment type. recommending a course of action. The task can provide. Therefore, the observations of what can be presented informally on-the-fly (e.g., a Assessment items must be designed around the informa- the student can do as demonstrated by his or question posed during a lecture), or it can be her performance on the assessment must be tion the user wants to obtain and the interrelationship formally prepared for class (e.g., test questions). applied to the model of cognition in order for the among cognition, observations, and interpretation that Tasks can involve “hands-on” performances assessor to draw inferences about the student’s forms the theoretical foundation of the item, but there or presentations, or projects. Ideally, students competence in the domain. also are a number of decisions that must be made about are given opportunities to demonstrate their the item’s structure and delivery to ensure that the item competencies with a variety of task types. Cognitive models can range from simple (e.g., a wrong serves the needs of the user most e&ciently and e"ec- answer means the student has an incomplete understand- 2. Response mode. Students’ responses to tively. !e primary considerations include characteristics questions or tasks can include selection from ing of the domain) to complex (e.g., each answer, right or of the item format, item quality, and cost-e"ectiveness options (multiple-choice), constructed-response wrong, provides insight into the student’s conception of of delivery and scoring. tasks that require writing a few sentences the domain or “mental model”). Sophisticated cognitive (short constructed answer) or a paragraph Item Characteristics models tend to have some empirical basis, often derived or more (extended constructed answer), oral from a combination of cognitive psychology and educa- Assessments, regardless of purpose, are composed of response (e.g., presentations), performance tion research. three components: a task, a response, and an evaluation of a task such as an experiment (performance

The Road Map Project | Assessment Report | Chapter 2 | Improving Teaching and Learning Through Assessment 32 of 75 Geography Executive Context and Purposes of Geography Example Appendices Preface Assessment Recommendations References Summary Goals Assessment Assessment Framework A, B, C Framework

assessments), a portfolio of students’ work, and Item Quality is measured and what we would like to measure. In so on. All of these modes can be used formatively It is important to recognize two issues about assessment: selecting the type of assessment format, issues of time in a classroom or for summative and program (1) any assessment is only a sample of what students and cost should be considered, while holding true to evaluation purposes. Selection of response the construct to be measured. Assessments designed to mode depends in part on what is being tested. know and can do out of a large universe of what might measure geographic practices, for example, are most Some formats tend to tap different cognitive be tested, and (2) any conclusions about what students likely to be problem-based performance tasks that processes: some are particularly useful for know and can do require interpretation mediated by a assessing factual recall (e.g., multiple choice), process of reasoning from evidence obtained from the entail multiple steps, making cost, time, and logistics while others are useful for observing students’ assessment; therefore, assessments are always imprecise considerable concerns. Students will likely be evaluated abilities with reasoning through explanatory to some degree (NRC, 2001). There are common based on fewer items due to the time and financial models (constructed response). Selection of principles that ensure the quality of assessments across costs associated with performance tasks. A mixture response mode also is influenced by the amount all item types. The two central principles are: of performance tasks that provides the opportunity of time that each response demands during to present an in-depth view of students’ thinking 1. Measure what is intended. An assessment an assessment, and the amount of resources and doing, with multiple-choice items that sample required to develop and score it. For instance, should be designed based on a clear idea of foundational competencies, often satisfies the need to an essay might provide extensive insight into what it intends to measure, or “the construct.” balance reliable and cost-effective measures. a student’s thinking about one question, but the The construct includes the content and/or the amount of time it takes to complete imposes practices that are being assessed. Articulation The Role of Frameworks in the limits on the amount of other concepts and skills of the construct gives focus to the assessment that can be evaluated during the assessment. development process and provides a benchmark Design of Assessments for verifying that the assessment will measure 3. Scoring system. A scoring system converts Because the creation of assessments that effectively what it is intended to measure. students’ responses to a numerical or other implement all of the design criteria discussed above is so 2. Meet technical requirements. Technical require- ordered scale. A scoring system is a form of challenging, educators have developed tools to guide the “interpretation model,” and the complexity of ments ensure that the information gathered using assessment design and development process. One of the the interpretation depends on the nature of the the assessment is valid (i.e., the assessment tests most powerful of these tools is the assessment frame- task, the response format and the context of the competencies it claims to test), reliable (i.e., the assessment. For example, selected response results are consistent), and fair (i.e., the assess- work. An assessment framework serves as a blueprint for items are generally scored as right or wrong ment does not favor a subset of the population assessment design and development, providing guide- (assuming there is only one correct answer). being tested). lines to assessment developers about how to construct Scoring is even more complex with constructed- an assessment. Assessment frameworks are generally response items, since the interpretation model Cost Effectiveness designed around a specific purpose, body of knowledge must account for all of the possible paths that The consideration of cost effectiveness requires or skills to be assessed, and population to be assessed. A students can follow to answer a question, and the assessment developers to recognize that the practical framework can play an important role in maintaining scoring rubric must take into account all possible reality that the ideal assessment may be too costly, time coherent goals for different assessments. For example, a correct responses along each of those paths. consuming, or logistically unwieldy to implement. framework can be used to construct both formative and Inevitably, compromises have to be made between what summative assessments for a course, ensuring that both

The Road Map Project | Assessment Report | Chapter 2 | Improving Teaching and Learning Through Assessment 33 of 75 Geography Executive Context and Purposes of Geography Example Appendices Preface Assessment Recommendations References Summary Goals Assessment Assessment Framework A, B, C Framework

are monitoring the same learning objectives. An as- Table 3. Framework Outline Using Science Content and Science Practices sessment framework also can play an important role Science Content in instructional design. In the approach commonly known as “backward design” (Wiggins & McTighe, Physical Science Content Life Science Content Earth and Space Science Statements Statements Content Statements 2005), instructional designers work backwards from

assessment goals to design instruction and formative Identifying science principles Performance expectation Performance expectation Performance expectation assessments. Using science principles Performance expectation Performance expectation Performance expectation An assessment framework plays multiple critical roles in the process of designing an assessment. A frame- Using scientific inquiry Performance expectation Performance expectation Performance expectation work conveys what knowledge, skills, and practices

should be assessed (and sometimes what should not) Practices Science Using technological design Performance expectation Performance expectation Performance expectation to those people responsible for creating a particular Source: Reproduced from NAEP Science Framework (NAGB, 2008). assessment, and a framework provides guidance Note: !is framework uses science content and science practices as the two dimensions of the framework. about the form the assessment should take, including Table 4. Framework Outline Using Geography Content and Geographic Cognition the types of items to use. An assessment framework provides a basis for validating the claim that the Content Dimension

assessments accurately reflect the learning goals they Physical Science Content Life Science Content Earth and Space Science are designed to assess, and a framework also provides Statements Statements Content Statements a means for maintaining consistency among assess- What mineral resources Where is the world’s largest What factors stimulate human ments developed by different people at different Knowing are often extracted by strip tropical rain forest? migrations? times and different places for different purposes. mining? Explain the motivations of Why are tropical rain forests Explain the effects of strip modern-day Mexicans and Typically, the core of an assessment framework is Understanding located near mining and shaft mining on Cubans for immigrating to the the equator? the landscape. a matrix organized around two dimensions. One United States. dimension, often called the content dimension, Compare current settlement How can both economic and describes the concepts and principles from the con- Support the conclusion that and employment patters environmental interest be Applying* tropical rain forests promote of Cuban and Mexican tent domain to be covered by assessments. The other, Dimension Cognitive reconciled in an area of strip wide species variation. immigrants in the United mining? often called the cognitive dimension, organizes behav- States. iors (i.e., what students should be able to do). The Source: Reproduced from NAEP Geography (NAGB, 2010). *Applying a range of higher-order thinking skills. names for the second dimension vary. For example, Note: !is assessment framework crosses content with cognition for the two dimensions of the framework. some assessment frameworks define the second di- 4 !ere is an important distinction between approaches to the cognitive dimension that is not always re%ected in the naming convention. In some mension as “practices” (Table 3), and others define cases, the cognitive dimension focuses on cognition (mental behaviors that are not observable), and in others, the cognitive dimension focuses on 4 external activities (observable behaviors). In some cases, the distinction is signaled by the names “cognition” versus “practices.” Regardless of the it as “cognition.” (Table 4). naming convention, it is important that assessment developers be conscious of the di"erence between cognitive and external behaviors, and that they outline goals for both.

The Road Map Project | Assessment Report | Chapter 2 | Improving Teaching and Learning Through Assessment 34 of 75 Geography Executive Context and Purposes of Geography Example Appendices Preface Assessment Recommendations References Summary Goals Assessment Assessment Framework A, B, C Framework

In the two dimensional matrix, a cell represents the In these examples, the framework is based on the to explain or predict where disagreements over the combination of specific content with a specific cogni- general assessment framework for geography presented division, control, and management of Earth’s surface tive behavior. Thus, the cells of the matrix inform test in Chapter 4. would occur. developers about what to assess. The contents of cells Item characteristic: Answers are multiple-choice with Example 1. The first example is for grades 3 to 5. The typically are called performance expectations (e.g., Tables map or other geographic representation. 3 and 4). performance expectation derives from the combination of a content statement about sources of conflict and the Figure 4 shows Item 1, which provides an example of In order for an assessment framework to guide assess- geographic practice of analyzing patterns on maps to an item that #ts these speci#cations. Item 1 displays ment developers, it needs to include information about generate explanations or predictions. This performance a map of a #ctitious region where natural resources the characteristics of an assessment, including the nature expectation, when combined with the desired multiple- are distributed across the territory of several nations, of the items to be included. This information can in- choice format represents a full specification of an item. including one oil #eld that is bisected by the border clude the types of items (e.g., multiple-choice, short an- between two nations, and the item asks students to Level: Grades 3 to 5 swer, performance assessment), the distribution of those identify a location that has the potential for con%ict in Content: items across content and practice areas (i.e., the relative Con%icts arise when there is disagreement that region. weight assigned to each performance expectation), and over the division, control, and management of Earth’s the levels of performance (e.g., below-basic, basic, pro- surface. !is content is from the second edition of Example 2. !e next example is targeted at high school ficient, advanced). As noted above, the specification of Geography for Life: National Geography Standards students. It shows how multiple performance expectations “item types” arises in recognition that assessment items (He"ron and Downs, 2012), (Standard 13, !eme 4, in a framework can be assessed together. In this case, the tapping a student’s ability to recall a fact or concept Grade 4). performance expectations derive from the intersection of a would be different from assessment items tapping a Cognition: Analyze geographic patterns on a map or single content category with three cognitive categories. !e student’s ability to design a well-controlled experiment other representation. content category concerns cooperation; and the cognitive or to use multiple documents to solve a problem. Put Performance expectation: Analyze patterns on a categories deal with knowing and understanding, answer- another way, different kinds of assessment items are map or other representation showing distribution of ing questions, and communicating. Below, we describe the needed to tap different performance expectations, and resources and borders or human settlements in order three sets of item speci#cations that are covered by Item 2. the assessment framework can organize the distribution of how items assess student abilities, in addition to what Table 5. Test-Item Weighting for Grade 12, Showing the Distribution of Items That Fall into Each the items assess (see Table 5). Category for the Grade 12 NAEP Geography Test Environment and Spatial Dynamics Content Cognition Space and Place Total To illustrate how assessment frameworks can be used Society and Connections to guide assessment development, we provide three Knowing 9 9 12 30 examples. In each of these examples, we show how content and cognitive dimensions can be combined Understanding 9 9 12 30 to describe a performance expectation, then we show Applying* 12 12 16 40

how a performance expectation and item characteristics Total 30 30 40 100 together can provide the specifications for an item. Source: NAEP Geography Framework (National Assessment Governing Board, 2010)

The Road Map Project | Assessment Report | Chapter 2 | Improving Teaching and Learning Through Assessment 35 of 75 Geography Executive Context and Purposes of Geography Example Appendices Preface Assessment Recommendations References Summary Goals Assessment Assessment Framework A, B, C Framework

Figure 4. Item 1: Identifying Potential Conflict in Level: Grades 9-12 Item characteristic: !e response takes the form of a the Region constructed response/essay. Specification 1: Item 1 Content: Cooperation between countries and organizations Item 2 (Figure 5) is an example of an item that could be may have lasting in%uences on past, present, and future used to assess these three performance expectations in a global issues (Geography for Life, Standard 13, !eme 3, constructed response format. It asks students to use their Grade 12). understanding of human cooperation to evaluate the long- Cognition: Explain a geographic principle or phenomenon. term impacts of two di"erent approaches to cooperation for Performance expectation: Explain how cooperation the United States, and to communicate their answers in the between countries and organizations may have lasting form of an argument made to a colleague. in%uences on past, present, and future global issues. Example 3. !e third example item also is designed Specification 2: for high school students. !is one focuses on physical Content: Cooperation between countries and organizations geography content—how the amount of incoming may have lasting in%uences on past, present, and future solar energy varies across the Earth’s surface—and global issues (Geography for Life, Standard 13, !eme 3, on the cognitive practices of organizing geographic Grade 12). information. Cognition: Evaluate one or more answers to a question or Content: Di"erent regions receive di"erent amounts Which two nations are most likely to have a con!ict over mineral resouces? solutions to a problem using geographic principles. A. Nation A and Nation B of solar energy during a year, and the amount of solar Performance expectation: B. Nation A and Nation C Evaluate one or more answers energy a place receives depends on the latitude and C. Nation A and Nation D to how cooperation between countries and organizations D. Nation C and Nation D cloud cover at that place. Source: 2001 NAEP Geography, Grade 4. may have lasting in%uences on past, present, and future Cognition: Sort data by spatial and other characteristics http://nces.ed.gov/nationsreportcard/itmrlsx/search.aspx?subject=geography global issues. to answer a geographic question. Performance expectation: Figure 5. Item 2: Cooperation between Countries Specification 3: Sort data by spatial and and Organizations Content: Cooperation between countries and organizations other characteristics to answer a question about the distribution of solar energy across a region. Item 2 may have lasting in%uences on past, present, and future You are a member of a Washington, DC, !ink Tank that gives Item characteristic: !e response takes the form of advice to the U.S. State Department. One of the issues you have global issues (Geography for Life, Standard 13, !eme 3, been considering is the extent to which the United States needs to a constructed response/representation. cooperate with other nations in order to be prosperous. Some Grade 12). members of the !ink Tank argue that since the end of the Cold Cognition: War, U.S. prosperity is much less dependent upon cooperation with Communicate using geographic principles, Item 3 (Figure 6) shows how this performance other nations. Others, however, maintain that cooperation among models, and data to educate or persuade an audience. nations, including the United States, is even more important today expectation might be evaluated in a computer- to promote prosperity. In which camp are you? Performance expectation: Persuade or inform an based assessment in which students use geographic In the space below, write a memo to the other members of the audience about how cooperation between countries and information systems (GIS) to organize geographic !ink Tank taking a position on the issue. Be sure to provide good reasons (evidence and examples) to justify your conclusions. organizations may have lasting in%uences on past, present, data in a way that prepares the student to answer a Source: Alternative Assessment in Geography, Grades 9-12. and future global issues. geographic question using the data. http://my.ilstu.edu/~jabraun/socialstudies/assess/geo/assess/9-12-13.html

The Road Map Project | Assessment Report | Chapter 2 | Improving Teaching and Learning Through Assessment 36 of 75 Geography Executive Context and Purposes of Geography Example Appendices Preface Assessment Recommendations References Summary Goals Assessment Assessment Framework A, B, C Framework

Figure 6. Item 3: Solar Power Task Implications for Improving ing assessment frameworks in our work. Assessment Geography Teaching and Learning frameworks can make the task of creating valid, reliable, Item 3 and useful assessments much more manageable, par- In this task, students are asked to identify locations appropriate E"orts to improve geography education can be ticularly when the target objectives are complex, such as for solar power generation in the United States. One step enhanced by attention to the design and use of assess- students must take to complete this task involves sorting solar reasoning and problem-solving skills. energy data for regions across the United States in such a way ments. Assessments for the purposes of informing to help them complete the task. !ey must use a GIS program instruction, measuring achievement, evaluating pro- !e Committee believes the creation of carefully de- to organize the data to show which states will be able to grams, and conducting research all can play important signed assessment frameworks for geography education generate the maximum amount of electricity over a year from their solar panels. roles in improving teaching and learning of geography. could play an important role in improving geography However, assessments must be carefully designed to align teaching and learning. We argue that a set of assessment A screen shot showing an example of a student response: with the learning goals, and they must be designed to frameworks tailored to the goals of geography educa- be sensitive to the desired information and the context tion would enable educators to develop assessments that in which the information will be used. An assessment would inform instruction, provide insight into program designed to help a teacher plan instruction for next week e"ectiveness, and build a research base that will ulti- will likely look very di"erent from an assessment cre- mately lead to substantial advancement of those goals. ated to evaluate the e"ectiveness of a newly introduced If successful, these assessment frameworks will improve instructional program. educators’ abilities to create and use assessments that: (1) !e design of assessments for geography should pay care- communicate the objectives of geography education to ful attention to each of the considerations described in teachers, students, and other stakeholders; and (2) yield this chapter. !e characteristics of assessments, item qual- the information that educators and students need to ity, and cost-e"ectiveness must all be chosen with the as- make suitable decisions about teaching and learning. sessment goals and purpose in mind to collect the desired In later chapters, we return to the challenges of designing information to assist geography teaching and learning. assessments and the role of assessment frameworks, Because the challenges of assessment design are consider- but #rst we turn to a discussion of the current state of Source: Adapted from Quellmalz et al. (2004). able, the Committee has primarily emphasized develop- assessment in geography education.

The Road Map Project | Assessment Report | Chapter 2 | Improving Teaching and Learning Through Assessment 37 of 75 Geography Executive Context and Purposes of Geography Example Appendices Preface Assessment Recommendations References Summary Goals Assessment Assessment Framework A, B, C Framework

Chapter 3: The Current Context for K–12 Geography Assessment

In this chapter, we consider the context in which this review of the most in%uential frameworks for assessment framework is intended to cover the breadth of report is responding: the current state of assessment in geography, and discuss their relationship to the goals geography knowledge and skills that would result from in geography education. As explained in the previous of geographic education established by Geography for good geographic instruction in the classroom. chapter, well-designed assessments o"er a mechanism Life. !is comparison allows us to evaluate how well for improving the e"ectiveness of education at scales existing assessment frameworks support progress toward !e NAEP geography framework is organized along from individual students and teachers to across districts those goals, and to identify opportunities that are not two dimensions (Table 4): a content dimension and a and nationwide. Because assessment frameworks ease currently being met by existing frameworks. cognitive dimension. !e content dimension is divided the challenge of designing e"ective assessments, the into three main areas: Space and Place, Environment Frameworks for Large-Scale Assessments Committee has made the creation of assessment frame- and Society, and Spatial Dynamics and Connections. works for geography education the focus of our work. !e three assessment frameworks that are most rel- Each of these content categories is explored in detail As part of this work, we found it necessary to investigate evant for K–12 geography in the United States are the within the framework, with speci#c objectives for each two questions about the current practice of assessment National Assessment of Educational Progress (NAEP) of the three grade levels. !e emphasis on content in geography education: What is the nature of existing assessment framework for geography, the Advanced is evident, and the relative lack of elaboration of the assessment frameworks for K–12 geography education? Placement (AP) Human Geography framework, and the cognitive dimension is equally evident. And, what is the nature of existing geography assess- NAEP assessment framework for science. Each is the !e cognitive dimension of the geography assessment ments? In this chapter, we present the results of these product of thoughtful e"orts to articulate educational is composed of three levels: Knowing, Understanding, investigations, #rst with an overview of currently used goals and each has been shaped by increasing attention and Applying (Table 4). Knowing (i.e., What is it? assessment frameworks. !en we provide the results of to measurement of learning outcomes and accountabil- Where is it?) assesses students on their ability to answer a study we conducted of existing assessment practices in ity over that same period. K–12 geography. We conclude the chapter with a dis- questions by recall. Understanding (i.e., Why is it there? National Assessment of Educational Progress in cussion of the needs and opportunities revealed by this How did it get there? What is its significance?) assesses Geography (1994, 2001, 2010). In the 1990s, Congress examination of recent history and current practices. students’ ability to provide explanations for phenomena. authorized the development of the broad-based NAEP !is cognitive level requires students to identify and Overview of Current Geography geography assessment at grades 4, 8, and 12. To support explain geographic patterns and processes. Applying Frameworks the development of this assessment, a committee of (i.e., How can knowledge and understanding be used to professional geographers, educators, and administrators solve geographic problems?) requires a range of higher- As discussed in Chapter 2, assessment can serve as a worked to create a framework that provided operational order thinking skills. At this cognitive level, students are means for improving geographic education. Our goal speci#cations for the geography assessments. !is expected to hypothesize, apply geographic principles to Guidelines for is to compare the actual practices of assessment with framework was heavily in%uenced by the new contexts, and form problem-solving models. the goals for geographic education as laid out in the Geographic Education (Joint Committee on Geographic second edition of Geography for Life: National Geography Education, 1984), though the #ve themes were reduced Together, the content and cognitive dimensions of the Standards (He"ron & Downs, 2012). We turn now to a to three categories of content. !e NAEP geography assessment form a matrix (Table 4) in which each con-

The Road Map Project | Assessment Report | Chapter 3 | The Current Context for K–12 Geography Assessment 38 of 75 Geography Executive Context and Purposes of Geography Example Appendices Preface Assessment Recommendations References Summary Goals Assessment Assessment Framework A, B, C Framework

tent area is measured at each cognitive level. !e cells of ability to synthesize their geographic knowledge using !e introduction of AP Human Geography has had the matrix describe example performances that charac- geographic skills and practices. a dramatic impact on the teaching of geography terize the intersection of the two dimensions, and assess- in American high schools. After several years of !ere is no traditional assessment framework for ment developers are given information about how much development, the course and test were launched in APHG available to the public. Instead, the teachers and each cell should be represented on an assessment (Table 2000-2001 with more than 3,200 tests administered. students are provided with a detailed course description. 5). !e framework proceeds to “drill down” to delineate Ten years later, more than 83,000 students took the While it takes the form of a course outline, rather in some detail each of the broad content areas involved. 2011 APHG exam. While the college-level exam than an explicit assessment framework, the description However, it does not have a comparable clari#cation of questions cannot be directly used to assess students’ of the APHG course plays the role of an assessment expectations for the cognitive (practices) dimension. ability to do high school geography, an increasing framework in communicating to teachers and students number of high school teachers model their course Assessment development is guided by three what students are expected to master. assessments on the constructed response and multiple- Achievement Levels (ALs) that are used to evaluate The APHG course outline divides content into choice questions published by the College Board. performance at each grade level: Basic, Pro#cient, and seven sections: Advanced. Basic indicates partial mastery of requisite NAEP Science (2009). In 2008, the National 1. Geography: Its Nature and Perspectives, knowledge and skills, Pro#cient represents solid Assessment Governing Board (NAGB) released a academic performance for each grade assessed, and 2. Population, major revision to the NAEP Science framework titled Advanced indicates superior performance. 3. Cultural Patterns and Processes, A Science Framework for the 2009 National Assessment of Educational Progress (NAGB, 2008; Fu, Raizen, & Advanced Placement Human Geography (2000). 4. Political Organization of Space, Shavelson, 2009). !e framework consists of a matrix Following the development of NAEP Geography, 5. Agriculture and Rural Land Use, (Table 3) with science practices on one dimension: advocacy by the geography education community 6. Industrialization and Economic Development, and identifying science principles, using science principles, and the Road Map Project partner organizations 7. Cities and Urban Land Use. using scienti#c inquiry, and using technological design. led the College Board to develop an AP course on To provide more details, each practice is expanded into Human Geography. !e Advanced Placement Human In the course outline, each of these sections is broken four general performance expectations. For example, Geography course (APHG) is not patterned on previous down into several subsections, each of which lists one of the performance expectations for “Using standards such as the Guidelines or Geography for Life, the topics to be covered in that subsection and the scienti#c inquiry” is to “conduct scienti#c investigations but rather re%ects a consensus of college faculty about percentage of the AP exam that will focus on that topic. using appropriate tools and techniques.” On the other the nature of modern human geography as taught dimension are the three components of science content: in colleges in the United States. Unlike the NAEP !e course outline also describes the following target physical science content statements, life science content geography assessment, the APHG exam covers only a practices and skills: use and think about maps and statements, and Earth and space sciences content portion of geography—human geography. spatial data, understand and interpret the implications statements, and each of these areas of content is further of associations among phenomena in places, recognize elaborated into learning objectives for elementary, !e two-part exam consists of a 60-minute multiple- and interpret at di"erent scales the relationships among middle, and high school students. choice section and a 75-minute open-response section. patterns and processes, de#ne regions and evaluate the !e #rst part focuses primarily on geography content, regionalization process, and characterize and analyze !e NAEP Science framework also includes examples but the second part is intended to assess students’ changing interconnections among places. of how a practice could be applied to a speci#c scienti#c

The Road Map Project | Assessment Report | Chapter 3 | The Current Context for K–12 Geography Assessment 39 of 75 Geography Executive Context and Purposes of Geography Example Appendices Preface Assessment Recommendations References Summary Goals Assessment Assessment Framework A, B, C Framework

context to create performance expectations that integrate sioned a study of existing assessments, both large-scale Once resources were selected from the collection, content and practices, as well as example items that and classroom-level. !e study was conducted by the those that did not have items that test geographic demonstrate how the performance expectation could be Committee research director under the direction of knowledge aligned to any of the six essential elements probed. !e description of the science practices and the the Committee co-chairs, and with assistance from from Geography for Life or to the geographic practices examples of how they could be used to assess scienti#c Committee members. outlined in Chapter 1 were excluded from further reasoning place far greater emphasis on integrating sci- analysis. When large-scale assessments were excluded, enti#c practices into NAEP Science assessments than the !e goal of the study was to shed light on which it was because they were general science or social NAEP Geography framework does currently. components of geography are being assessed currently studies assessments that did not contain any items with in grades K–12 and how those components are being geographic content. When classroom resources were !e framework’s focus on practices is evident in the assessed. To conduct this study, we collected publicly excluded, it usually was because they did not contain 2009 NAEP Science assessment, which includes a new released items from large-scale assessments and any assessment items at all. Out of 114 randomly set of hands-on interactive computer assessments to classroom assessments from commercially published selected resources in the collection, 79 of the resources assess students’ ability to perform the practices. !ese and widely distributed non-commercial instructional were determined to have items that were testing computer tasks provided students with a simulated materials. Our speci#c objectives were to: (1) compare geographic competencies, and these resources yielded environment to apply their scienti#c skills and practices, what is being assessed with the goals for geography 696 geography items for analysis. giving them a more open-ended environment than stu- education described in Geography for Life, and (2) dents had been provided in previous NAEP assessments. compare how geography is being assessed against the !e study categorized items according to: (1) targeted !e rich, interactive environment allows students to criteria for assessment discussed in the previous chapter. ability—what is being assessed, (2) item characteristics— test hypotheses and design and execute experiments, !is analysis of current assessments provides empirical how the item assesses student competencies, and providing insight into students’ ability to reason with data for the recommendations we make in the chapters (3) confounding factors—whether the item is likely scienti#c evidence. In practice, these items have revealed that follow. to accurately re%ect what students know. !ese three information about students that traditional content- criteria were used as the basis for a taxonomy (Table 6) Overview of Methods5 focused multiple-choice and short-answer NAEP that was used to code each item. assessments had not. In a special report on the results !e study is based on an analysis of items from of these hands-on interactive assessments, the National Targeted ability describes the substance of what the resources that were selected at random from our entire Center for Education Statistics (2012) showed that just item assesses. It includes the geographic content, geo- resource collection. !e full resource collection included over one-half of high school students reached the cor- graphic skills, and the type of cognition that the item assessments that were labeled as geography assessments, rect conclusions for the scienti#c investigations on the requires. !ese criteria provide the core information as well as assessments from other areas of science and exam, but fewer than 30 percent of those students could about which competencies are being assessed. Content social studies that were judged to be likely to include justify their conclusions satisfactorily. is classi#ed according to the standards from Geography items assessing geographic knowledge and practices. for Life, geographic skills are classi#ed using categories !e resource collection included both large-scale and Current Geography adapted from Geography for Life by the Committee, and classroom-level assessments. !e large-scale assessments Assessment Practices cognitive demand is classi#ed according to the catego- included state, national, and international assessments. ries described by Li, Ruiz-Primo, & Shavelson (2006). To obtain a picture of current assessment practices in Classroom assessments were drawn from textbooks, K–12 geography education, the Committee commis- geography units, and online item banks for teachers. 5 See Appendix B for additional details about the methods for the study.

The Road Map Project | Assessment Report | Chapter 3 | The Current Context for K–12 Geography Assessment 40 of 75 Geography Executive Context and Purposes of Geography Example Appendices Preface Assessment Recommendations References Summary Goals Assessment Assessment Framework A, B, C Framework

Table 6. Geography Item Coding Taxonomy !e item characteristics category captures the way in in which the substance of the item was inaccurate, in- which the assessment task was presented to the students. cluding if additional answer choices could be considered Coding categories Subcategories Under item characteristics, we recorded information correct. Finally, we described ways that students could 1. Geographic content about the setting in which the assessment was used, use non-geographic knowledge to evaluate the item, 2. Geographic skills and the targeted grade level, how the item is structured, such as using logic to eliminate unlikely answer choices. Targeted ability practices and what type of graphical representations appear in Findings 3. Cognitive demand the stem or answer choices, if any. Coding for these Geography Content Coverage 4. Assessment setting characteristics enabled us to look for patterns in how goals are being assessed. 5. Grade band Our #rst analysis examined the distribution of content Item characteristics 6. Item format !e confounding factors category includes problems with covered across the items. Our objective was to see if Geography 7. Type of representation the way the item is written that make it an inaccurate assessment items are aligned to the content in for Life. 8. Clarity (or less accurate) measure of what students know and 9. Content accuracy can do. !e confounding factors for which we collected 6 Confounding factors We found items assessing all #ve of the content-focused 10. Vulnerability to test- information include whether the way the item is written essential elements in Geography for Life. !is distribution taking strategies could make it di&cult for students to understand what con#rms that at the coarsest level of analysis, current was being asked and whether the language was unneces- assessment practices are supportive of the goals for sarily complex. We also recorded whether there are ways content coverage expressed by Geography for Life.

Figure 7. Frequency Distribution of Assessment Items in Large-Scale Assessments and Classroom When we examined the relative frequency of speci#c Assessments for the Content-Focused Essential Elements from Geography for Life essential elements, however, a pattern of uneven Large Scale Classroom distribution emerged (Figure 7). Across all large-scale 40% and classroom items, two categories, Environment and Society and Places and Regions, were assessed by ems 30% substantially fewer items than the others. When we looked only at assessments that were explicitly labeled 20% as geography assessments (excluding science and other social studies assessments), we observed that

entage of It 10% the three middle categories (see Figure 7) are assessed rc more evenly, but !e World in Spatial Terms is even Pe 0% more heavily emphasized relative to the others, and world in places and physical systems human systems environment Environment and Society slightly less. spatial terms regions and society 6 !ere are six essential elements in Geography for Life, but the sixth is Geographic Content explicitly about the application of geographic knowledge, so we do not consider it to be content-focused.

The Road Map Project | Assessment Report | Chapter 3 | The Current Context for K–12 Geography Assessment 41 of 75 Geography Executive Context and Purposes of Geography Example Appendices Preface Assessment Recommendations References Summary Goals Assessment Assessment Framework A, B, C Framework

When we compared the content distribution in large- properties of maps, Earth’s physical processes, and the practices from assessments to be consistent across grade scale assessments with classroom assessments, di"erent characteristics of speci#c places on Earth. Certainly, not levels. In fact, middle school items have the highest patterns of emphasis emerged (Figure 7). Most notably, all standards demand the same amount of attention, but percentage of items that require a geographic practice items aligned to Physical Systems are overrepresented this uneven distribution is cause for concern if it signals (40%). !e heavy emphasis on analyzing information in large-scale tests and underrepresented in classroom to teachers that some standards are unimportant, or if may re%ect the fact that it is regarded as the easiest assessments. In addition, while Environment and Society it indicates that educators are not receiving information practice to assess in a multiple-choice or short-answer they should have about the status of their students’ is underrepresented overall, that content area is assessed format. For example, the most common type of analysis learning in certain areas. even less frequently in classroom assessments than in in this item involved students observing a graphic, large-scale tests. Geographic Skills and Practices7 such as a photograph or a map, and characterizing the !e large number of assessments targeting Physical As discussed in Chapter 1, since the 1960s, major pattern that they see. See Figure 9 for an example of an Systems in large-scale tests is likely a result of that e"orts to communicate the goals of geographic educa- item that probes students’ ability to look for patterns category’s overlap with earth science, a topic that more tion have argued for the importance of by comparing settlements and features of the physical frequently appears in large-scale tests compared with geographic skills and practices, but these Figure 8. Frequency Distribution of Large-scale Geography other geography content. However, this content area arguments have had limited impact on Assessment Items that Target Each Geographic Practice rarely appears in geography assessments at the classroom the classroom. Not surprisingly then, our level. Moreover, we #nd the underrepresentation of study reveals that only 30% of geog- 80% items aligned to Environment and Society in classroom raphy assessment items required that 70% assessments worrisome, given the mounting importance students use any geographic practices at

ems 60% for students to understand humans’ interactions with all (Figure 8). A clear pattern emerged their environment. It is possible that this topic is from this analysis, showing that most of 50% assessed infrequently because it spans the boundary those items assess the same geographic 40% between social studies and science, and demands practice, analyzing geographic informa- tion. Analyzing geographic information 30%

competence with both disciplines. Rather than being entage of It 20% addressed in complementary ways by each discipline, is assessed in 21% of large-scale items, rc

it seems that the category is being overlooked by both. but the other geographic practices rarely Pe 10% appear in any assessment items at all. 0% When we considered coverage of speci#c standards One might expect that items asking e s s within the sixteen essential elements, we found the students to apply their knowledge to a non aphic aphic distribution to be even more uneven. We found that geographic question or problem might ganizing question aphic data information ommunicating 40% of all items evaluated knowledge from only ering questionsc be reserved for high school assessments, w posing geogr geogr ans aphic information three standards. !ese three standards focus on the but we #nd that absence of geographic analyzing geogr llecting and or co and designing solutiongeogr 7 In the analysis for geographic skills and practices and all subsequent analyses, we report only data for large-scale assessments because of the challenges of conducting random sampling with classroom materials. See Appendix A for more information. Geographic Practices

The Road Map Project | Assessment Report | Chapter 3 | The Current Context for K–12 Geography Assessment 42 of 75 Geography Executive Context and Purposes of Geography Example Appendices Preface Assessment Recommendations References Summary Goals Assessment Assessment Framework A, B, C Framework

Figure 9. An Example of Middle School Social Studies Item that Has a environment. Other practices, such knowing how, when, and where knowledge applies). !is Representation (Map) and Probes a Geographic Practice (Analysis) as data collection, perhaps seem last #nding was anticipated, because an item requir- more di&cult to evaluate in a format ing schematic knowledge often would require probing The map below shows a coastal region with a river winding through the mountains and meeting the coastline with a delta. Early agricultural sites found in this region are that easy to score. competencies in multiple content areas and practices, indicated with white dots. which makes assessment design much more complex. Our #ndings indicate that geograph- Both factual knowledge and the ability to read maps ic practices are not being assessed in and graphs are important elements of geography lit- a way that conveys their importance. eracy. !e minimal demand for higher-order thinking As a result, current assessments are and reasoning with geographic concepts and the overall providing an incomplete picture of imbalanced pattern of assessment across cognitive de- students’ pro#ciency with geography. mands, however, requires increased attention to support Cognitive Demand assessing knowledge and skills that better re%ect the To get a sense of the depth of reasoning we want students to be prepared to do with conceptual understanding being their geographic knowledge. assessed, we analyzed items for Item Characteristics the cognitive demands required to According to this map, most early agricultural sites in this region developed near the answer the questions (Figure 10; see A. mountains. !e dominant item format used in both large-scale and B. rivers. Table A4 for a description of cogni- classroom geography items is selected response, with C. coastlines. D. deltas. tive demands). We found that almost 68% of all items using some form of multiple-choice 54% of the items tapped students’ format (Figure 11). Classroom assessments o"er more declarative knowledge (i.e., knowing Figure 10. Frequency Distribution of the Cognitive Demand Required diverse opportunities for students to represent their that for Large-Scale Geography Items ), often at the level of recognizing knowledge; 47% of classroom items use short answer or a de#nition. More than 27% assessed essay formats, though few classroom materials include 60% students’ procedural knowledge (i.e., scoring rubrics, which limits their utility (and assess- 50% knowing how), which includes read- ems ment developers would not consider open-response 40% ing and gathering in-formation from items questions without a rubric). Fewer than 3% of maps, graphs, and texts. Only 17% 30% all items ask students to make or modify a representa- of geography items require schematic tion, a format that holds promising opportunities for 20% knowledge (i.e., knowing why), which evaluating students’ geographic reasoning. entage of It 10% includes explaining an unfamil- rc iar context by drawing on general We also looked at the use of representations that

Pe 0% geographic principles or models. students must refer to in order to answer a question. declarative procedural schematic strategic Virtually none of the items directly Representations appeared in 59% of the items, with Cognitive Demand tapped strategic knowledge (i.e., maps being the most common (Figure 12). However,

The Road Map Project | Assessment Report | Chapter 3 | The Current Context for K–12 Geography Assessment 43 of 75 Geography Executive Context and Purposes of Geography Example Appendices Preface Assessment Recommendations References Summary Goals Assessment Assessment Framework A, B, C Framework

we found that only 16% of the questions with maps Figure 11. Frequency Distribution of Item Formats for Classroom and Large Scale Geography Items required students to process the information on the Large Scale Classroom map, as does the item in Figure 9. On the other hand, 90% 84% asked students to read information directly from 80% the map, indicating that map reading is well represented in geography assessments, but reasoning with geospatial 70% ems representations is not. 60%

Confounding Factors 50%

We found that 60% of the items displayed apparent 40% problems that could impede students’ ability to re%ect entage of It 30%

their knowledge. One of the most common problems rc

found in our analysis, particularly for large-scale tests, was Pe 20% ambiguous questions and/or answers (14%). Even more 10% common is that at least one answer choice is implausible (15%). !e concern with any confounding factors such as 0% these, is that they raise doubts as to whether the results of multiple choice true/false matching short answer paragraph/ representation essay those geography assessments are valid indicators of what Item Format students know or are able to do. Figure 12. Frequency Distribution of Types of Representations that Appear as Part of a Large Scale Implications for Improving Geography Geography Assessment Item Education 50%

Taken together, the discussions in this chapter present 40% a compelling picture of the needs and opportunities for ems assessment in geography education. 30% Needs 20% Our review of current assessment practices exposes weak- entage of It

nesses in the way geography is being assessed across the rc 10%

curriculum. !ese weaknesses include: Pe

• Coverage of content areas is uneven. Most content 0% standards are underrepresented in assessments. none illustration map/globe graph/table diagram document This trend signals to educators and learners that this Representation Types

The Road Map Project | Assessment Report | Chapter 3 | The Current Context for K–12 Geography Assessment 44 of 75 Geography Executive Context and Purposes of Geography Example Appendices Preface Assessment Recommendations References Summary Goals Assessment Assessment Framework A, B, C Framework

content is less important and it indicates that then it does not provide the information about questions, problems, and phenomena. !is bias toward educators are not being provided with informa- students’ understanding and abilities that educa- the lower-level knowing and understanding portion of tion about student learning in these areas. tors need to make good instructional decisions. cognition is clearly re%ected in the assessments that are • Assessments tend to focus on less-demanding We believe that addressing these weaknesses is an being used to evaluate students’ geographic literacy. cognitive tasks. The overwhelming majority of essential step toward improving geography education assessment items in our survey focus on recall- Geography for Life describes a vision of geographic overall, and our recommendations in the chapters that ing or recognizing geographic facts and simple education in which students use spatial analytical skills follow are intended to help educators to do so. procedural knowledge, such as using a map to reason about people, places, and connections among legend to retrieve information from the map. Stu- Opportunities for Improvement them, but we #nd that assessments present a much dents rarely are asked to apply their geographic During the course of nearly 50 years, geographers narrower view of expectations for geographic literacy. knowledge to a novel context, explain or justify !erefore, we see an enormous opportunity to use their reasoning, or evaluate the strength of an ar- and educators have produced increasingly explicit assessment as a mechanism to shift geography education gument. The result is that these assessments rein- descriptions of the goals of geography education, closer to the version of geographic literacy captured by force a conception of geography as being about culminating in the second edition of Geography for Life. Geography for Life. Assessments that increase attention facts and simple recall; they also fail to provide !is vision includes expectations for students by grade educators with an accurate picture of students’ band and example performances for knowledge and to the roles of geographic reasoning and problem ability to reason, analyze, or communicate about understanding, perspectives, and skills. While educators solving, that draw on the full range of geographic questions and problems with their geographic and curriculum developers have adopted the general concepts, and that challenge students to manipulate knowledge. goals for knowledge and understanding, it appears their knowledge with higher cognitive demands could • Technical quality can be improved. A substantial that they have not adopted the goals for skills and provide a mechanism to communicate these priorities number of items in our survey have structural practices to the same degree. With the exception of the to teachers and students, and such assessments would or content flaws that threaten their validity as 2009 NAEP science framework, our review of existing provide tools to help educators monitor learning. In measures of student understanding and ability. frameworks for large-scale assessment indicates that the chapters that follow, we describe an approach to If an item is difficult to understand, has inaccu- these assessments are acting to maintain the existing assessment that uses a framework organized around rate content, or allows students to narrow down focus on content, rather than signaling the importance these priorities, speci#cally designed to capitalize on answer choices through test-taking strategies, of applying content knowledge to exploring geographic these opportunities for improvement.

The Road Map Project | Assessment Report | Chapter 3 | The Current Context for K–12 Geography Assessment 45 of 75 Geography Executive Context and Purposes of Geography Example Appendices Preface Assessment Recommendations References Summary Goals Assessment Assessment Framework A, B, C Framework

Chapter 4: A 21st Century Assessment Framework for the Geographical Sciences

Chapter 3 revealed that existing geography assessments dimension is made up of knowing, understanding, and but the contents of each dimension and, therefore, do not re%ect the priorities established by the Geography geographic practices. !e cells in the body of the matrix the constructs developed from them, will vary. !e for Life: National Geography Standards (He"ron & are designed to hold performance expectations, which framework dimensions are speci#ed based on the Downs, 2012). !e same three (out of 16) content describe the application of a cognitive behavior to a goals for the population that will be assessed. !e standards are assessed in 40% of the items, competency geographic concept or principle. goals will change in scope, detail, and sophistication. with geographic practices is rarely assessed, and For example, an elementary school geography course AFGS21 is designed to support assessment of critical structural problems with items are common. Here, that focuses speci#cally on human geography might knowledge and practices for geography. Specifically, it is we present an assessment framework that we believe assess only content within Human Systems, and would designed to: will help assessment developers create assessments eliminate the other content areas from the framework. that support the overall goals of geography education 1. support the development of high-quality assess- Similarly, this course might target only a subset of the geographic practices, so the practices that students by providing a structure to de#ne, organize, and ments that are matched to the goals of the unit, would not be expected to use would be eliminated systematically select the assessment goals. We call this course, standard or program being assessed; and from the framework. Because of its concentration on framework the 21st Century Assessment Framework for 2. facilitate the development of assessments that one content area, the course might require far more the Geographical Sciences (AFGS21). demand integrated knowledge, understanding, detailed understanding of that content than a broader skills, and practices. Our goal is that this assessment framework serves course, so there could be more content and cognition as a blueprint to support the development of a new AFGS21 has been designed as a general assessment goals speci#ed under Human Systems than included generation of assessment frameworks, and they will framework, meaning that it is not intended to be used in a framework for another course. In addition, the generate assessments with higher #delity to the goals directly to generate specific assessments. Rather, it is description of expectations for competency for each of the content and cognition goals would be much less for geographic literacy and the principles of assessment designed to serve as a blueprint that developers can sophisticated than in a framework of the same scope for development. In this chapter, we describe the process of follow for developing specific assessment frameworks a comparable high school course. using the AFGS21 to create speci#c frameworks. for specific purposes. By using AFGS21 as a guide, developers of assessments can achieve the objectives The Content Dimension in AFGS21 A 21st Century Assessment that AFGS21 was designed to achieve, applied to their Framework for the Geographical assessment context. !e content dimension of AFGS21 enumerates and Sciences describes the geography concepts and principles to A “specific assessment framework” is used by an be assessed. In AFGS21, the content dimension is AFGS21, like most contemporary assessment assessment developer to create instruments tailored segmented into 16 categories corresponding to 16 of frameworks, is based on a 2-dimensional matrix with to the particular set of goals for the population being the 18 standards from Geography for Life.8 Within a “content” dimension and a “cognitive” dimension assessed (see Chapter 5 for an example of a speci#c (Figure 13). !e content dimension is made up of assessment framework). !e process of using the 8 Of the 18 standards, two (Standards 17 and 18) focus on activities. !ey have been incorporated into the practices portion of the geographic concepts and principles, and the cognitive framework to create assessments does not change, framework’s cognitive dimension.

The Road Map Project | Assessment Report | Chapter 4 | A 21st Century Assessment Framework for the Geographical Sciences 46 of 75 Geography Executive Context and Purposes of Geography Example Appendices Preface Assessment Recommendations References Summary Goals Assessment Assessment Framework A, B, C Framework

AFGS21 Matrix

Figure 13. Preview of the 21st Century Assessment Framework for the Geographical Sciences.

Download the AFGS21 Matrix at http://education.nationalgeographic.com/ media/file/AFGS21_Matrix.xlsx

The Road Map Project | Assessment Report | Chapter 4 | A 21st Century Assessment Framework for the Geographical Sciences 47 of 75 Geography Executive Context and Purposes of Geography Example Appendices Preface Assessment Recommendations References Summary Goals Assessment Assessment Framework A, B, C Framework

each of these categories, there are subcategories for The Categories of Geographic Table 7. The Six Categories of Geographic Practices in AFGS21 the speci#c content statements that Geography for Life Practices in AFGS21 and the Geographic Practices That Fall Within Those Categories provides for each of the standards. Depending on the While it is important for students to master Practice Category Geographic Practices standard, there are between one and four subcategories the foundational facts and concepts of a. Identify problems or questions that per standard. geography, the geographic advantage comes 1. Posing geographic can be addressed using geographic from the ability to apply this understand- principles, models, and data; express Because the framework describes the culmination of the questions problems and questions in geographic K–12 curriculum, we used the content statements for ing to answer meaningful questions and terms. solve meaningful problems. !e central grades 9-12 to de#ne the content subcategories, but one a. Identify geographic data that can could construct versions of AFGS21 using elementary goal of 21st century geography education help to answer a question or solve a problem. school or middle school content statements from is to prepare students to be able to reason 2. Acquiring Geography for Life. through questions and problems that they geographic b. Collect data (incl. observations or will encounter in their personal, profes- information measurements) about geographic phenomena, and/or gather existing The Cognitive Dimension in AFGS21 sional, and civic lives. data to help answer a question or solve a problem. !e cognitive dimension of AFGS21 describes expecta- !e geography practices in AFGS21 3. Organizing a. Organize data and create tions for performing geographic thinking and reasoning. describe the kinds of activities students geographic representations of data to help solve a AFGS21 divides cognition into two major categories: should be pre-pared to perform in order to information problem or answer a question. reason through geographic questions and 1. knowing and understanding geographic a. Identify data analysis strategies that principles and phenomena, and problems. We divide these practices into six can be used to help solve a geographic categories, each of which encompasses a set problem or answer a question. 2. geography practices. of more speci#c practices (Table 7). 4. Analyzing b. Find and describe spatial and temporal patterns in data, or find data that Knowing and Understanding in AFGS21 geographic While these practices are presented as a lin- matches a pattern, to help answer a information question or solve a problem. !e #rst category in AFGS21’s cognitive dimension ear sequence, this should not be interpreted is “knowing and understanding.” From an assessment as indicating that they necessarily proceed c. Construct an explanation or prediction perspective, knowing and understanding corresponds for phenomena by comparing data to a in this order in practice, or that they should. principle or model. to the ability to identify, describe, and explain facts and A geography investigation might start with concepts. !e ultimate goal of geography education a. Construct an answer to a question or a any one of these practices, and it may even solution to a problem using geographic is to cultivate the ability to perform complex reason- 5. Answering require only one of them. Alternatively, principles, models, and data. ing, but knowing geographic facts and understanding questions and it might move iteratively back and forth designing b. Evaluate one or more answers to a geographic concepts is critical for that reasoning. !is question or solutions to a problem between one or more steps. As Figure 14 solutions means that assessments must probe knowing and un- using geographic principles, models, indicates, the requirements of the speci#c and data. derstanding to accurately characterize students’ progress question or problem and its context should toward geographic literacy. 6. Communicating a. Communicate using geographic data, determine which practices are necessary and with geographic principles, and models to educate or in what order they should be performed. information persuade an audience.

The Road Map Project | Assessment Report | Chapter 4 | A 21st Century Assessment Framework for the Geographical Sciences 48 of 75 Geography Executive Context and Purposes of Geography Example Appendices Preface Assessment Recommendations References Summary Goals Assessment Assessment Framework A, B, C Framework

We also note that the term data in the descriptions of practices in AFGS21 and the practices as they have and involve humans and/or their environment. of the speci#c practices includes both qualitative and been described in recent science and geography educa- Geographic data are integral to geographic in- quantitative data. tion standards and framework documents is illustrated quiry because they provide information about the current or historical state of the world that can in Table 8. Five categories of geographic practices were derived be used to understand people, places, and the from the geographic skills described in Geography for The Geographic Practices in AFGS21 connections between them. A key aspect of geo- graphic data is their association with a specific Life, with one addition, communicating with geographic !e speci#c practices in AFGS21 describe the ways location; for data to be geographic, they must information. !is addition makes the framework consis- that geographic knowledge and understanding can be have a spatial component. Given the association tent with the categories of scienti#c practices described applied to answer questions and solve problems. While with a location, geographic patterns are particu- in the National Assessment of Educational Progress these practices have analogues across the natural and larly visual when displayed across a map, and (NAEP) 2009 Science framework and the NRC (2012) social sciences and engineering, they are distinctively these visualizations are useful for analyses as Framework for Science Education. !e categories of prac- geographic and, as such, contribute to the geographic well as for communication. tices speak directly to the nature of geography, but they advantage. For example: • Applying geographic theories and models also have a clear relationship to the model of inquiry • Posing and answering geographic questions. to data. The use of geographic principles models, that underlies recent assessment and standards frame- The data that geographers work with often theories, and data might be characterized as works for science. !e relationship among the categories overlap with those from other fields; for example, the “geographic perspective.” Geographic historians also consider people and places, principles, theories, and models are components of geographic content that, when appropriately Figure 14. A Graphical Representation of the economists look at patterns within and among applied to data, have explanatory and predictive Relationship Between the Geographic Practice human settlements, and ecologists study living power. These principles focus primarily on Categories and the Geographic Question or and physical systems. Geographic questions spatial relationships between people and places, Problem Driving the Inquiry Process differ from other types of questions in that and they can be used to make sense of patterns Acquiring they may look at the same data, but from a Geographic Information in geographic data, answer questions, and spatial perspective. The posing of a geographic construct solutions. question might involve looking at the same Analyzing Answering data the economist is looking at, but the spatial In Table 9, we present some examples of speci#c Geographic Questions Information and focus might inspire a question that leads the geographic practices from AFGS21. !ese tasks can be Designing geographic investigation to incorporate different Solutions used as part of instruction and assessment. Geographic variables, analyses, and ultimately, to contribute Question or Problem different information to understanding a Using AFGS21 to Develop Specific phenomenon. Assessment Frameworks Posing Communicating • Collecting, organizing, and creating visualiza- Geographic with Geographic Questions Information tions of geographic data. As with any scientific Earlier, we explained that, as a general assessment discipline, geography is focused on understand- framework, AFGS21 is designed to serve as a blueprint Organizing ing phenomena. In geography, these are phe- for the development of targeted assessment frameworks Geographic Information nomena at or near the surface of the Earth, for speci#c purposes. !e Committee’s intent is that

The Road Map Project | Assessment Report | Chapter 4 | A 21st Century Assessment Framework for the Geographical Sciences 49 of 75 Geography Executive Context and Purposes of Geography Example Appendices Preface Assessment Recommendations References Summary Goals Assessment Assessment Framework A, B, C Framework

AFGS21 will give rise to a variety of speci#c frameworks assessments for teachers to use to guide and pose will lead them to create di"erent frameworks. !e for assessments that measure students’ abilities to inform instruction, bene#t of AFGS21 is that o"ers each user a process for engage in complex reasoning and geographic practices. • researchers designing instruments to evaluate the developing frameworks and a structure that will enable efficacy of one or more educational curricula or Our objective is to serve a wide range of assessment programs, them to create frameworks that remain faithful to the developers, including: core content and practices of AFGS21. • district or school committees developing end- • committees developing nationwide or statewide of-course exams to assess individual student !e development of a speci#c geography assessment assessments to inform policy and practice, achievement, and framework from AFGS21 is a 4-step process: • state and district committees developing • classroom teachers developing, refining, and 1. specification of content goals, assessments for school-level accountability, aligning courses to revised state standards using the “backwards design” approach to instructional 2. specification of cognition goals, • classroom resource (e.g., textbooks and design (see Wiggins & McTighe, 2005). instructional materials) developers creating end- 3. identification and clarification of performance of-unit exams for use by teachers, Each of these audiences will bring di"erent constraints expectations, and • classroom resource developers creating formative and goals to the use of AFGS21, and their particular pur- 4. specification of assessment characteristics.

Table 8. Comparison Between Geographic Practices in the AFGS21 and Geography for Life, and Scientific Practices as They Have Been Conceptualized in Recent Standards and Framework Documents

21st Century Framework for Geography for Life, 2nd NRC Framework College Board Science ACT College Readiness the Geographical Sciences Edition (2012) for Science Education (2012) Standards for Success (2009) StandardsTM (2010) (2012) Asking questions (science) Posing geographic questions Asking geographic questions Scientific questions defining problems (engineering) Scientific investigation Planning and carrying out Acquiring geographic information Acquiring geographic information Generation of evidence investigations

Organizing geographic information Organizing geographic information Analyzing and interpreting data Data analysis Interpretation of data

Analyzing geographic information Analyzing geographic information Developing and using models Quantitative applications

Constructing explanations (for science) and designing solutions Evidence-based explanations and Evaluation of models, inferences, Answering questions and Answering geographic questions (for engineering) models and experimental results designing solutions Engaging in argument from evidence Communicating with geographic Obtaining, evaluating, and information communicating information

The Road Map Project | Assessment Report | Chapter 4 | A 21st Century Assessment Framework for the Geographical Sciences 50 of 75 Geography Executive Context and Purposes of Geography Example Appendices Preface Assessment Recommendations References Summary Goals Assessment Assessment Framework A, B, C Framework

!e primary product of this process is an assessment produces concrete descriptions of the knowledge needed Framework developers #rst need to consider the matrix with content descriptors across the columns, to have pro#ciency with the content objective, and purpose of their framework to determine which general cognition descriptors down the rows, and performance these descriptions indicate the boundaries and the level practices they need to probe. It is not necessary that expectations in the cells. !e process also results in a set of sophistication at which students may be assessed on a framework should include each of the geographic of speci#cations for assessment developers on how to that knowledge. practices, but it is important that the decision to include generate assessments from the matrix, and some drafts or exclude practices from an assessment framework be !e product of Step 1 is a list of content knowledge and of items as models to guide item development. made deliberately. !e framework developers must ask: concepts that can be used to label the columns in an What does the targeted population of students need to Step 1: Specification of Content assessment framework. be able to do with their geographic knowledge? What !e #rst step in developing a speci#c assessment Step 2: Specification of Cognition kinds of geographic questions and problems should framework from AFGS21 is the identi#cation and !e second step involves considering the goals for the they be able to investigate? What geographic practices description of the content goals for assessment. !e cognitive dimension: specifying what students should would be required for those investigations, and which framework developer9 must consider the purpose know and understand, and the geographic practices of those practices should students be assessed on? Based of the assessment to generate explicit objectives for they should be able to perform. For example, taking on the answers to these questions, they should select the content. In many cases, the objectives at the the term “analyze” from the Virginia standard above, the required rows of the general AFGS21 framework to coarsest level will be pre-determined. For example, the assessment developer must consider: What would incorporate into their speci#c framework. the development of a statewide assessment typically be a satisfactory analysis of trends for the purpose of starts with state standards, and the development of !e #rst level of geographic practices in the AFGS21, the assessment? Should students be able to perform an end-of-course exam typically starts with course the categories of practices, is described at a general, statistical calculations? Or are observations and objectives and a course outline. However, these abstract level. To target precisely the practices that are descriptions of patterns su&cient? Are there certain objectives may be articulated in a way that combines necessary and su&cient for investigating the geographic kinds of trends in migrations or cultural interactions content with cognition. For example, the Virginia state questions and problems of interest, the practices must that students should be able to analyze? standards include the following standard for world be described more explicitly. For that reason, the geography: !e student will analyze past and present !e process of elaborating content objectives is Committee has developed a set of prompts to assist trends in human migration and cultural interaction a familiar one to those involved with designing framework developers in generating a detailed set of as they are in%uenced by social, economic, political, assessments. !e process of specifying the #rst level of geographic practices (Table 10). and environmental factors (Virginia Department of geographic cognition, knowing and understanding, also !ese prompts pose questions about which speci#c Education, 2008). In this case, the objective would need has been explored in detail. By contrast, elaborating geographic skills, practices, questions, and data are ap- to be revised to extract the content goal. In addition, geographic practices is much less common. Descriptions propriate at di"erent stages of learning, and the amount objectives generally are expressed too vaguely, or are too of practices to be assessed as detailed as we advocate of structure that a learner at that stage might require. broadly conceptual for use in developing assessments, here are even more rare, so we provide a thorough !e prompts about structure are based on the model so they must be “unpacked” to describe explicit goals explanation of how the AFGS21 can serve as a model of a learner’s zone of proximal development (Vygotsky, (example in Chapter 5). !is unpacking process for completing this step. 1978), in other words, the range of tasks that a learner 9 In this report we distinguish creators of assessment frameworks from creators of assessments. Framework developers create assessment is capable of doing with a certain amount of external frameworks. Assessment developers use assessment frameworks to build assessments to be administered to students.

The Road Map Project | Assessment Report | Chapter 4 | A 21st Century Assessment Framework for the Geographical Sciences 51 of 75 Geography Executive Context and Purposes of Geography Example Appendices Preface Assessment Recommendations References Summary Goals Assessment Assessment Framework A, B, C Framework

Table 9. Examples of Kinds of Tasks That Can Be Used to Demonstrate Competency with Geographic Practices

Practices Example Tasks

a. Identify problems or questions that can • Describe geographic problems and questions that can be addressed using a observations, measurements, geographic concepts, and be addressed using geographic principles, models models, and data; express problems and • Recognize whether or not questions and problems are scientifically investigable using geographic information. Posing Posing questions in geographic terms. questions • State questions and problems using geographic terminology and ways of thinking geographic geographic

a. Identify geographic data that can help to • Plan how to collect data via both primary and secondary sources answer a question or solve a problem. • Determine which methods of data collection are most appropriate in addressing a given question or problem

b. Collect data (incl. observations or measure- ments) about geographic phenomena, and/ • Describe the ways in which and the places where data can be collected Acquiring Acquiring geographic geographic information or gather existing data to help answer a • Collect original data as well as data from secondary sources question or solve a problem.

• Recognize/describe ways to organize or represent data a. Organize data and create representations • Organize data or create an original representation of data to help solve a problem or answer a question • Explain why a type of representation will or will not help in addressing a question Organizing Organizing

geographic geographic • Compare the effectiveness of organizational structures or representations information

• Identify and describe the methods with which data can be analyzed a. Identify data analysis strategies that can be • Describe the qualitative and quantitative analyses and appropriate strategies for solving the problem used to help solve a geographic problem or answer a question. • Generate or select parameters for querying data to find a location • Explain which parameters are/aren’t helpful in solving the problem

• Comparing, classifying, finding, and describing relationships, associations and trends in data b. Find and describe spatial and temporal • Characterize the data patterns (i.e. linearity, clustering, dispersion, etc.) patterns in data, or find data that matches a • Extrapolate from a pattern in data in order to infer how the pattern extends in (i.e. the past, present, other geographic regions) pattern, to help answer a question or solve a problem. • Apply parameters to a data set and describe how well the data for different places match the desired criteria • Look at data to indicate places with matching attributes

c. Construct an explanation or prediction for • Develop explanations for phenomena by comparing relationships and patterns in data to geographic principles and models phenomena by comparing data to a prin- • Compare data as prompted by a given principle, theory or model. Analyzing geographic information geographic Analyzing ciple or model. • Describe the extent to which data fit the generalizations, theories and models

a. Construct an answer to a question or a • Answer a question/solve a problem using an analysis of the relevant data to justify the answer solution to a problem using geographic principles, models, and data. • Explain how an answer is supported by a given data representation depicting a pattern or a trend

• Evaluate a response to a geographic question, a solution to a geographic problem, or an opinion about a geographic issue b. Evaluate one or more answers to a question • Compare an argument to data to determine the extent to which data supports it solutions or solutions to a problem using geographic • Describe information they would use to evaluate the argument and designing principles, models, and data. • Compare several arguments and explain why they are/aren’t supported by the data Answering questions questions Answering • Discuss whether given data is enough to support a given argument

- • Address an audience by writing about, making a representation of, or orally presenting an argument or solution to a geographic problem a. Communicate using geographic data, prin- • Describe the information they would include based on their audience ciples, and models to educate or persuade an audience. • Compare different examples of communication and infer who they think the audience and the examples’ effectiveness for the given

Communi audience cating with cating geographic geographic information

The Road Map Project | Assessment Report | Chapter 4 | A 21st Century Assessment Framework for the Geographical Sciences 52 of 75 Geography Executive Context and Purposes of Geography Example Appendices Preface Assessment Recommendations References Summary Goals Assessment Assessment Framework A, B, C Framework

Table 10. The Geographic Practices with Prompts for Creating a Specific Framework

Practices Example Tasks

a. Identify problems or questions that can be addressed using geographic principles, • What specific kinds of problems and questions are appropriate for students at this stage? models, and data; express problems and • How much support and structure do students at this stage require to formulate a question or problem? Posing Posing questions in geographic terms. questions geographic geographic

a. Identify geographic data that can help to • What data are appropriate for students at this stage? answer a question or solve a problem.

• What forms of observation and measurement techniques are appropriate for students at this stage? b. Collect data (incl. observations or measure- • Are students capable of designing a data collection protocol independently, or do they require structure and support? ments) about geographic phenomena, and/ • What sampling strategies are appropriate for students at this stage? Acquiring Acquiring or gather existing data to help answer a geographic geographic information question or solve a problem. • What kinds of archives for existing data are appropriate for students at this stage? • How much structure and support do students need to locate and assemble data from existing archives?

a. Organize data and create representations • Which types of maps and other data representations are students prepared to construct and interpret? of data to help solve a problem or answer a • How refined should the data be before students are asked to organize it? question. • Should students be able to choose how to organize the data, or are they just expected to be able to organize it in a given way? Organizing Organizing geographic geographic information

a. Identify data analysis strategies that can be used to help solve a geographic problem or • What data analysis techniques should students at this stage know about and be able to perform? answer a question.

• Which kinds of patterns should students be able to identify? b. Find and describe spatial and temporal patterns in data, or find data that matches a • What forms and representations of data should students be able to read for patterns? pattern, to help answer a question or solve a • Which processes for identifying patterns in data are appropriate? problem. • How much structure do they require for conducting the identification/description process?

• What kinds or forms of data should students be able to compare against explanations or predictions? c. Construct an explanation or prediction for • Are there certain inconsistencies between explanations for a phenomenon and data for that phenomenon that students should be able to phenomena by comparing data to a principle identify? or model. • How structured should this comparison be for students? Analyzing geographic information geographic Analyzing

a. Construct an answer to a question or a • Are there specific kinds of questions and problems should students be able to respond to? solution to a problem using geographic prin- • How structured should the response be; should students be expected to use the data and generate a response? Should they be given a ciples, models, and data. response and asked to support it with data?

b. Evaluate one or more answers to a question • In what ways should students be able to judge how well evidence supports an argument? solutions or solutions to a problem using geographic

and designing principles, models, and data. • How much structure should they be given in their evaluation and comparisons of alternative arguments? Answering questions questions Answering

• In what formats should students be expected to communicate? - a. Communicate using geographic data, • Would students be judged on how well they communicate as well as the appropriateness of the mode of communication that they choose principles, and models to educate or or the quality of the supporting information? persuade an audience. • Should students be able to select the appropriate supporting information or should that be given to them? Communi cating with cating geographic geographic information • Are there certain audiences students should be prepared to communicate an argument or solution to?

The Road Map Project | Assessment Report | Chapter 4 | A 21st Century Assessment Framework for the Geographical Sciences 53 of 75 Geography Executive Context and Purposes of Geography Example Appendices Preface Assessment Recommendations References Summary Goals Assessment Assessment Framework A, B, C Framework

structure or support. For example, at one moment in a main. Each of the cells is #lled with at least one assess- expectations in the cells. process of learning and development, a learner might able description of what students should know and be Step 4: Specification of Assessment be able to interpret spatial patterns on a map with the able to do. Depending on the breadth of content in the Characteristics assistance of a sequence of prompting questions. At a column and the range of cognition in the row, there can In the fourth step, for each performance expectation later stage, students might be expected to complete the be a large number of possible performance expectations framework developers specify types of items, including same task without any external support or structure. in each cell. !e decision about how comprehensive the cognitive demands and item formats. !e cognitive So, the framework developer must consider the stage of cell should be depends on the purpose of the assess- demands (see details in Chapter 3) describe the degree learning that is appropriate to expect for the population ment. A framework that will be used to generate forma- to which students are expected to manipulate informa- of students that will be assessed. tive and summative assessments, for example, might require several performance expectations for each cell. tion to complete an item. For example, the framework To develop an assessment framework tailored for a Performance expectations always should be reviewed to developers could designate that an item probing a certain speci#c purpose, framework developers may use these ensure that they are clear, concrete, and speci#c enough cell should draw on declarative knowledge, whereas an prompts and their understanding of the audience to to provide su&cient guidance for assessment developers. item probing another performance should test students’ create speci#c descriptions of practices to be assessed. procedural knowledge. !ere could be more than one Framework developers might #nd a di"erent set of Once the performance expectations have been written, item probing the same performance expectation, but each prompts more suitable for describing the expectations framework developers should provide guidance to assess- item tests students’ competency at a di"erent cognitive for their particular population, so they should not be ment developers on how to select performance expecta- level. Most importantly, there should be a range of cogni- limited to this list of questions, but they could consider tions to use in developing items. In most cases, there tive demands speci#ed across the framework matrix. will be more performance expectations in a framework it a model for creating their own prompts. than can be practically assessed, so framework develop- Likewise, framework developers should consider the nature of the performance expectation and cognitive !e product of Step 2 is a list of practices that are used ers should indicate to item writers how to sample across demand to designate an item format for each cell that to label the rows in the assessment matrix. the matrix in a manner that would re%ect their priori- ties. !is can be communicated by attaching a weight to will be assessed to ensure that students have a variety Step 3: Identification and Elaboration of of opportunities to demonstrate their competencies. In Performance Expectations each performance expectation or section of the matrix to signal how often items that probe those cells should deciding item formats, the following factors should be In the third step, framework developers use the intersec- be represented, as in Table 5. Alternatively, framework considered (see Table 2): the purpose of the assessment, tions of the two dimensions to systematically de#ne a developers can simply select which cells are the priorities the construct measured, practical constraints that the set of performance expectations that describe the ap- for the item developers, which is the approach used in assessment developers will be operating under, the con- plication of practices to content. the example framework in Chapter 5. In that example, text in which the assessment will be administered and speci#c cells on the matrix are highlighted for developers scored, the student population that will be assessed, and !e description of performance expectations is cre- to indicate that the performance expectations in those the way the assessment information is to be used. !ese ated by #lling in the cells in the body of the assessment cells should be assessed. factors will guide the speci#cations in the development matrix with a statement that combines the content from process to ensure that a variety of formats are being used the column and the cognition from the row. When the !e product of Step 3 is a completed assessment matrix across the matrix, and ideally, that students are given the cells are all #lled in, they encompass the full range of for the speci#c assessment framework together with opportunity to be assessed on any performance using the performances that could be assessed across the do- guidance on how to sample from the performance multiple forms.

The Road Map Project | Assessment Report | Chapter 4 | A 21st Century Assessment Framework for the Geographical Sciences 54 of 75 Geography Executive Context and Purposes of Geography Example Appendices Preface Assessment Recommendations References Summary Goals Assessment Assessment Framework A, B, C Framework

In addition to specifying item types, we recommend that Implications for Improving the design of high-quality assessments that advance the framework developers also provide sample items to assess- Geography Education goals of geography education. When used in this way, ment developers to ensure that they are communicating the framework serves as a blueprint for frameworks that their intent. !ese samples should illustrate the approach As a general framework, the primary role for AFGS21 is can serve these purposes in a concrete and practical way. that framework developers recommend, including format, to communicate what should be assessed and how an as- !ese speci#c frameworks will convey to assessment sessment can be built around those objectives. Its central cognitive demand, and other characteristics. Providing developers the importance of balance among know- message is that the goal of geography education is a bal- samples helps to give assessment developers an understand- ing, understanding, and engaging in practices and the ing of the kind of information that the framework devel- ance of the main components in the cognitive dimension: integration of practices with content that are embodied opers want the assessments to collect about the students. knowing, understanding, and engaging in practices. in AFGS21’s structure. To illustrate what this might look !e product of Step 4 is a speci#cation of the kinds Beyond its communicative function, the framework has like in practice, we provide an example of the process of items that assessment developers should use in the a practical role: to support the creation of frameworks of developing a speci#c framework using AFGS21 in construction of the assessment, as well as prototype items targeted to speci#c purposes, which in turn, will guide Chapter 5. that #t these speci#cations.

The Road Map Project | Assessment Report | Chapter 4 | A 21st Century Assessment Framework for the Geographical Sciences 55 of 75 Geography Executive Context and Purposes of Geography Example Appendices Preface Assessment Recommendations References Summary Goals Assessment Assessment Framework A, B, C Framework

Chapter 5: An Example Geography Assessment Framework

In the previous chapter, we introduced a general assess- ing, and in some cases, policy of geography (Shavelson be designed around assessing those standards. !ey also ment framework for geography, the 21st Century Assess- et al., 2008). Assessment experts often can manage the were encouraged to integrate assessment of Virginia ment Framework for the Geographical Sciences (AFGS21). technical aspects behind what needs to be in a frame- standards for English and mathematics if appropriate. We describe it as a general framework because it is too work, but capturing the essential learning goals requires Prior to the development team beginning their work, we broad for practical use; it encompasses the entire set of an understanding of the discipline that only a content clari#ed how the assessment framework they were being content standards from the second edition of Geogra- expert can bring. In turn, a content expert can ensure asked to create might be used by a school or district. !ey phy for Life: National Geography Standards (He"ron & accuracy to the discipline, but an educator also must be were told to approach their task as if their framework Downs, 2012) and all of the geographic practices. !e involved to check that expectations are reasonable and was going to be used as part of an explicit e"ort to move purpose of AFGS21 is to serve as a blueprint or tem- appropriate for the students. Finally, for large-scale as- instruction beyond ”knowing and understanding” to plate for speci#c assessment frameworks with narrower sessments district or state administrators are required to include geographic practices. So, we asked them to think scope of content and practices. In Chapter 4, we also bring an understanding of how the assessments will be about their assessment framework as part of a strategy for described a procedure for creating speci#c frameworks used within the educational system. communicating the importance of teaching geographic for a targeted set of learning objectives and audiences In the development of the assessment framework pre- content and how that content can be used to answer from the general AFGS21 framework. In this chapter, sented here, we assembled a team consisting of experts 10 questions and solve problems using geographic practices. we give an example of that process and its products. in geography and assessment, classroom geography We also told the team that the framework and the To generate this example, the Committee convened a teachers, and district and state-level administrators. !e team worked together to outline goals, create a performance expectations in the framework should working group with the charge of creating an assessment framework, and describe speci#cations for item develop- support the creation of both formative and summative framework from AFGS21 for a speci#c purpose and ers. !e team was led by the Assessment Committee’s assessments for teachers and, potentially, the design of context. !e process was set up to follow as closely as research director. large-scale assessments for districts or states. As part of possible the process that a working group from a school developing this framework, we asked them to create district or state agency would follow. In the remainder Preparing the Team sample assessments. Speci#cally, we requested examples of the chapter, we describe the entire process from of formative assessments for use during project-based determining the composition of the working group, !e framework development team was given the task instruction and examples of summative assessments that to creating the assessment matrix, to creating sample of developing an assessment framework for a six-week could be used as part of an end-of-unit or end-of-year assessment items. high school unit on human settlement. !ey were asked assessment of student’s overall understanding of content to undertake this task as if the framework was to be and practices from the unit. Assembling the Framework used for developing assessments to be used in several

Development Team di"erent settings across a district or a state. To make 10 !is chapter includes excerpts of the example framework and the task concrete, they were told that the unit would drafts of assessments derived from the framework. See Appendix B !e construction of an assessment framework requires for the entire example framework and drafts of formative and cover a cluster of high school social studies standards summative assessments designed to be models for assessment a team with expertise in the content, assessment, teach- from Virginia, so their assessment framework should developers to follow.

The Road Map Project | Assessment Report | Chapter 5 | An Example Geography Assessment Framework 56 of 75 Geography Executive Context and Purposes of Geography Example Appendices Preface Assessment Recommendations References Summary Goals Assessment Assessment Framework A, B, C Framework

Finally, the development team was told that this example c) mapping, describing, and evaluating the Standards for 12th Grade English was not intended to serve as a practical assessment formation of economic unions. 12.1 The student will make a five- to ten-minute formal framework—in part because that would require more WG.11 The student will analyze the patterns of urban oral presentation. extensive review, and in part because we would not have development by a) Choose the purpose of the presentation: to time to #ll in all of the columns. Instead, it would be a) applying the concepts of site and situation to defend a position, to entertain an audience, an illustration of the process of generating a speci#c major cities in each region, or to explain information. assessment framework based on the generic framework, AFGS21. b ) explaining how the functions of towns and b) Use a well-structured narrative or logical cities have changed over time, and argument. !e speci#c standards that the development team was c) describing the unique in%uence of urban areas c) Use details, illustrations, statistics, comparisons, given to work from were from the Virginia 12th grade and some challenges they face. and analogies to support purposes. Social Studies Standards (Virginia Department of Skills for World Geography d) Use visual aids or technology to support Education, 2008): WG.1 The student will use maps, globes, satellite images, presentation. Standards for World Geography (WG) photographs, or diagrams to e) Use grammatically correct language, including WG.5 The student will compare and contrast the distri- a) obtain geographical information about the vocabulary appropriate to the topic, audience, bution, growth rates, and characteristics of human world’s countries, cities, and environments; and purpose. population in terms of settlement patterns and the 12.2 The student will evaluate formal presentations. location of natural and capital resources. b) apply the concepts of location, scale, map projection, or orientation; a) Critique relationships among purpose, WG.7 The student will identify types of natural, c) develop and re#ne mental maps of world regions; audience, and content of presentations. human, and capital resources and explain their significance by d) create and compare political, physical, and b) Critique e"ectiveness of presentations. thematic maps and 12.4 a) showing their in%uence on patterns of economic The student will read and analyze a variety of infor- e) analyze and explain how di"erent cultures use mational materials, including electronic resources. activity and land use, and maps and other visual images to re%ect their b) evaluating perspectives and consequences own interests and ambitions. a) Identify formats common to new publications regarding the use of resources. and information resources. WG.12 The student will apply geography to interpret WG.9 The student will analyze the global patterns and the past, understand the present, and plan for b) Recognize and apply specialized informational networks of economic interdependence by the future by vocabulary. c) Evaluate a product based on analysis of the a) identifying factors, including comparative a) using geographic knowledge, skills, and accompanying warranty and instruction manual. advantage, that in%uence economic activities and perspectives to analyze problems and make trade; decisions and d) Evaluate the quality of informational and technical materials. b) describing ways that economic and social b) relating current events to the physical and interactions have changed over time and human characteristics of places and regions.

The Road Map Project | Assessment Report | Chapter 5 | An Example Geography Assessment Framework 57 of 75 Geography Executive Context and Purposes of Geography Example Appendices Preface Assessment Recommendations References Summary Goals Assessment Assessment Framework A, B, C Framework

12.8 The student will write documented research papers. framework. !e categories they selected are indicated Figure 15. Content Categories for a High School with an asterisk. Unit on Human Settlement a) Identify and understand the ethical issues of research and documentation. With the content categories de#ned, the next step is to *1. Resource distribution b) Evaluate the accuracy and usefulness of elaborate them in the form of statements that describe 2. Safety and security information. in more detail what students will need to know and *3. People design places c) Synthesize information to support the thesis. understand. Because these are descriptions of content, *4. Systems of interconnections d) Present information in a logical manner. not cognition, these content statements take the form *5. Characteristics of people and places of principles, concepts, and models. To identify the 6. Urban change e) Cite sources of information, using a standard content statements for their framework, the team posed method of documentation, such as that of the questions about what concepts, principles, and models Modern Language Association (MLA) or the Figure 16. An Example of a Content Statement are essential for the kind of reasoning about human American Psychological Association (APA). Under the Resource Distribution Content Category settlements that students might encounter around them for the Example Framework f) Edit copies for correct use of language, spelling, or in the news. For example, the team articulated the punctuation, and capitalization. concept in Figure 16 to describe that set of relationships g) Proofread #nal copy and prepare document for that students should know and understand to be able The resources needed to fulfill human needs are unevenly distributed across publication or submission. to address questions and problems around resource space and over time, which influences Developing the Framework distribution. the location, size, and spatial organization of settlements, and requires movement of Step 2: Selection and description of cognition !e framework development process took place in resources and people between places. categories. Once the content categories have been de- two parts: matrix development and speci#cation of #ned in explicit statements, the next step is to select and assessment characteristics. de#ne the categories of cognition to be assessed. Unless Figure 17. An Example of a Set of Geographic Part 1. Developing the Assessment Matrix the assessment framework is being used exclusively to Practices Specified for the Cognition Dimension Step 1: Selection and description of content develop items that require practices, then the “knowing” of the Example Framework categories. !e #rst step in creating an assessment and “understanding” categories always are included in 4b. Find and describe spatial and matrix from AFGS21 is selecting the content to be a framework. For the example framework, the develop- temporal patterns in data, or find data ment team considered all 12 of the geographic practices covered and categorizing it into the columns that that match a pattern, to help answer organize the content dimension. Drawing on the team’s under the six practice categories, and they described in a question or solve a problem. understanding of the domain and experience with detail what they thought high school students should Students find patterns in data including teaching these topics to high school students, the team be able to do with geographic data by the end of a six- clusters and nodes, edges and boundar- week unit on human settlement. !ese practices are not proposed that the framework should include the six ies, population pyramids, anomalies to categories of content as shown in Figure 15. Because an exhaustive list of everything that could be assessed; patterns and arrangements, gradation, time was limited, the team was asked to select only they represent what the group, based on their expertise, size, hierarchy, sequences, and order four of the six themes to %esh out for the example decided were high priorities for geographic reasoning across space.

The Road Map Project | Assessment Report | Chapter 5 | An Example Geography Assessment Framework 58 of 75 Geography Executive Context and Purposes of Geography Example Appendices Preface Assessment Recommendations References Summary Goals Assessment Assessment Framework A, B, C Framework

around the topic of human settlement. For example, tion. !e team went through the same process with Figure 18. Performance Expectations for the Example Figure 17 shows that for practice 4b under “Analyze the understanding category. Again, they started with Framework Aligned with Resource Distribution Geographic Information,” the development team speci- the concepts and translated their components into as- (Content) and Knowing and Understanding (Cognition) #ed the following set of spatial and temporal patterns sessable statements where students have to describe a that high school students might look for to answer geo- complex system of relationships or explain how or why Identify or describe a geographic principle graphic questions and to solve problems around human a phenomenon occurs. For example, Figure 18 shows or phenomenon settlements. the performance expectations describing what students • Identify factors that influence resource distribu- should be able to identify or describe to have the knowl- tion. !e process of specifying the content and cognition edge they will need for resource distribution. • Identify SEPE (social, economic, political, envi- dimensions is necessarily iterative; as the team moved ronmental) factors that influence how and where to the next step and started considering speci#c !e performance expectations in the Knowing and Under- people settle. phenomena and what students would need to do standing rows will be used as the objectives for assessing • Give examples of resources that are unevenly with that phenomena to answer geographic questions students’ competence with the content. !e performanc- distributed. and solve problems, they continually returned to the es in these two rows require only content knowledge, • Describe ways that settlements use resources. descriptions of content and cognition to make sure whereas all performances in the rest of the matrix ask • Describe ways that settlements acquire resources.

that all of the knowledge, skills, and practices students students to apply elements of this content to geographic • Identify resources needed by settlements. would need were represented. Underlying all decisions questions and problems using geographic practices. • Identify relationships between settlements and were questions about what is appropriate for high Step 4: Description of performance expectations— the environment. school students, what is reasonable for a six-week unit, geographic practices. • Identify ways that the human capital impacts and what are realistic expectations in terms of resources To describe performance Geographic Practices a settlement. available to public school classrooms. expectations for the rows, the team crossed the conceptual model in the content dimension • Describe decisions that settlements make Step 3: Description of performance expectations— with the practice in the cognition dimension; in the cell about resources. know and understand statements. Once the two where they intersect the team described something a • Describe ways that resource distribution can dimensions of the matrix have been laid out, the next student could be expected to do that would show that change over time. step is to begin describing performance expectations. the student could use both the content and the practice • Identify reasons why access to resources can !e team started specifying performance expectations to answer a geographic question or solve a problem. be limited. by focusing on the #rst two rows, the Knowing and !e team used the speci#c Knowing and Understanding Understanding statements. !ey began with the con- statements and speci#c skills and practices to generate an Explanation or Prediction in the practices dimension, cepts and conceptual models that had been described these concrete descriptions. Team members also brain- they describe performance expectations (Figure 19). for the content dimension in Step 1, and they translated stormed phenomena that they would like students to its components into assessable knowing statements. have experience reasoning through, and they used that !e development team articulated the performance !e team considered everything students would need as a context to help identify appropriate, realistic, and expectations in such a way that any performance to know to recognize, identify, de#ne, or describe to engaging performances. For example, under Resource could be assessed with a variety of di"erent contexts. show that they know each concept and how they func- Distribution on the content dimension and Construct !erefore, any performance could be assessed multiple

The Road Map Project | Assessment Report | Chapter 5 | An Example Geography Assessment Framework 59 of 75 Geography Executive Context and Purposes of Geography Example Appendices Preface Assessment Recommendations References Summary Goals Assessment Assessment Framework A, B, C Framework

times in multiple ways, providing deep insight into the Figure 19. Performance Expectations for the Example Framework for Resource Distribution (Content) and students’ competence with each objective. Furthermore, Construct an Explanation or Prediction (Cognition)

multiple items aligned to the same performance • Compare trends of changes to resource demands by a settlement that has been studied to the resource de- expectation could require di"erent cognitive demands, mands of a settlement with similar characteristics to make a prediction about how demands may change in the or could use di"erent item formats, each providing future. distinct information about a students’ ability to reason • Evaluate how well a pattern of settlement could be explained by dependence on a resource using models of around that content and practice. how different populations depend on resources based on economic factors, social factors, cultural factors, or environmental factors. Part 2. Specifying Assessment • Evaluate implications of different decisions on how to distribute access to a resource by comparing each option Characteristics to an appropriate model of how allowing or limiting access to that resource affects a population, such as com- paring how allowing open enrollment to a school has affected the students, the community around the school, Once the development team had #lled in all of the per- and communities in the region. formance expectation cells (see the #nished framework in Appendix B), the team turned its attention toward achievement. characteristics of items for the assessment. !ey were high school class might investigate to illustrate how the !e achievement assessment group given two di"erent purposes for assessment: (1) one was framework could be used to structure project-based evaluated the entire matrix, and discussed priorities for to be a formative assessment, designed to elicit student learning and performance assessments around the a 45-minute long end-of-unit test. !e group decided thinking with each of the selected performances, such performance expectations described in the framework. that Practices 1 (posing questions) and 6 (communicating) that the teacher could diagnose if each student was !e project is designed to provide students with would be more e"ectively assessed through embedded developing the desired competency, and if not, where experience applying their knowledge of concepts around performance assessments, such as an end-of-project the student was getting confused; and (2) the other resource distribution using some of the geographic evaluation of students’ products, so those rows were assessment was to be a summative assessment of student practices, such as identifying questions that can be eliminated from consideration for their assessment. !e achievement, designed to sample student competen- addressed using geographic principles, models and data, group then selected a set of cells from the matrix to use cies at a broader scale across the whole matrix. !ese and expressing those questions in geographic terms. for item development, at least two from each row and items would evaluate how well the student mastered !e group then designed prompts for the teacher to use each column to ensure that each part of the content and the performance expectations. Although the items to check students’ progress toward competency with cognition dimension was being assessed in more than would span a wide range of di&culty so the teacher can the performance expectation throughout the project, one way (Table 12). capture students’ who lie on the upper and lower ends and a scoring rubric to evaluate their responses. Figure !e group then created a smaller matrix consisting of the spectrum of mastery, the items would not need to 20 shows a task given to students as part of the project only of the cells they aimed to target with this test, and provide detailed information about the students’ incor- in which students propose a geographic question that the group added item development recommendations rect thinking. !e team split into two groups, one for they could investigate in an e"ort to solve a geographic about item characteristics (Table 12). !is distribution each assessment. problem. !e activity provides information to help ensures that the test will cover the range of item formats teachers evaluate how well their students are mastering Step 5a: Development of a project-based learning and cognitive demand across the practices and content, this practice (see the rubric illustrated in Table 11). and performance assessment. !e formative allowing students opportunities to demonstrate their assessment group described a speci#c problem that a Step 5b: Development of an assessment for competencies in di"erent ways and at di"erent levels. A

The Road Map Project | Assessment Report | Chapter 5 | An Example Geography Assessment Framework 60 of 75 Geography Executive Context and Purposes of Geography Example Appendices Preface Assessment Recommendations References Summary Goals Assessment Assessment Framework A, B, C Framework

Table 11. Scoring Rubric for Formative Assessments Embedded in a Project Aligned with the Example Framework

Skill Developing Proficient Advanced

Limited participation in brainstorming of possible geographic Actively participates in brainstorming of possible geographic Takes leadership role in brainstorming of possible geographic questions questions questions

Poses questions related to the overarching project that may Poses geographic questions related to the overarching proj- Poses multiple, investigable, relevant geographic questions Posing Posing questions

geographic geographic not be geographic, investigable, or relevant ect some of which are investigable or relevant related to the overarching project

Loosely connects the data to the question and assesses the Connects the data to the questions and assesses the quality Thoroughly connects the data to the questions, and assesses quality of the data for the question (obvious gaps are pres- of the data for the question (minor gaps may exist) the quality of the data for the question (no gaps are present) ent)

May not identify parts of questions that are addressed by ex- Identifies some parts of questions that are not addressed Identifies all parts of questions that are not addressed by Acquiring Acquiring geographic geographic information isting data; specifies some data, not necessarily geographic, by existing data; specifies geographic data to address those existing data; specifies a variety of geographic data and to address those questions questions sources to address those questions

Identifies existing theories with which to construct an expla- Identifies appropriate existing geographic theories with Identifies a range of appropriate existing geographic theories nation (may not be appropriate or geographic) which to construct an explanation with which to construct an explanation

Constructs an explanation that fully compares interpreta- Constructs a limited explanation; may not fully compare inter- Constructs an explanation that compares interpretation of tion of the data to the range of possible existing theories or pretation of the data to existing theories or models the data to existing theories or models models Organizing Organizing geographic geographic information Evaluates how well the theory or model explains the data Evaluates how well the theory or model explains the data Evaluates how well the theory or model explains the data (no (obvious gaps are present) (minor gaps may exist) gaps are present)

May or may not determine if the initial geographic guiding Determines if the initial geographic guiding question needs Determines if the initial geographic guiding question needs to question needs to be refined or revisited and, if needed, par- to be refined or revisited and, if needed, updates the question be refined or revisited and, if needed, thoroughly updates the tially updates the question and analysis and analysis question and analysis

Constructs answers to the question that are consistent with Constructs answers to the question that are consistent with Constructs thorough answers to the question that are con- the data. Uses the explanations constructed to support the the data; uses the explanations constructed to support the sistent with the data; uses the explanations constructed to answers (obvious gaps are present) answers (some gaps may exist) support the answers (no gaps exist) information Uses the answers to propose a partial solution to the over- Uses the answers to propose a solution to the overarching Uses the answers to propose multiple solutions to the over-

Analyzing geographic geographic Analyzing arching geographic problem geographic problem. arching geographic problem.

Partially identifies the level of detail and types of representa- Completely identifies the level of detail and types of repre- Identifies the level of detail and types of representations ap- tions appropriate for the intended audience (obvious gaps sentations appropriate for the intended audience (no gaps propriate for the intended audience (some gaps may exist) are present) exist)

Produces and delivers an effective and geographically sup- Produces and delivers a convincing and geographically sound

solutions Produces and delivers an incomplete presentation communi- ported presentation communicating possible solutions to the presentation communicating possible solutions to the over-

and designing cating possible solutions to the overarching problem overarching problem (minor gaps may exist) arching problem Answering questions questions Answering

Partially answers questions from the audience by referring to Answers questions from the audience by referring to data, Comprehensively answers questions from the audience by data, analysis and constructed representations analysis, and constructed representations referring to data, analysis, and constructed representations Communi- cating with cating geographic geographic information

The Road Map Project | Assessment Report | Chapter 5 | An Example Geography Assessment Framework 61 of 75 Geography Executive Context and Purposes of Geography Example Appendices Preface Assessment Recommendations References Summary Goals Assessment Assessment Framework A, B, C Framework

Figure 20. An Excerpt from a Project Aligned test with items that sample the domain with this scope also wanted to ensure they were conveying their vision to the Example Assessment Framework with of content, practices, cognitive demands, and item of how the items based on this framework might look. Embedded Formative Assessment formats also provides teachers a depth of information To that end, the group brainstormed several item that can be used evaluate how well students mastered outlines based on the framework speci#cations that the topic and that can be used to pinpoint areas of item developers could use as guides for realizing the weakness. group’s conceptualization of items (Figure 21). !ese Step 6: Development of sample item outlines. !e descriptions are just conceptions of what an item group recognized that the framework is a result of aligned to the framework might look like. Although extensive conversations about how the high school the descriptions are incomplete and they have not been students could be assessed on their development of validated, they serve as another type of recommendation knowledge and skills for this unit. !e framework to help item developers interpret the intent of the captures the essence of these discussions, but the group assessment framework experts.

Table 12. Matrix with Specifications for Item Developers for the Distribution of Items for a High School End-of-Unit Exam on Human Settlement

Resource People Design Systems of Characteristics of Distribution Places Interconnections People and Places

Knowing and selected selected understanding response response

Posing geographic questions

Acquiring geographic selected constructed information response response

Organizing geographic constructed selected information response response

Analyzing geographic constructed selected constructed information response response response

Answering questions constructed selected and designing response response solutions

Communicating using geographic information

The Road Map Project | Assessment Report | Chapter 5 | An Example Geography Assessment Framework 62 of 75 Geography Executive Context and Purposes of Geography Example Appendices Preface Assessment Recommendations References Summary Goals Assessment Assessment Framework A, B, C Framework

Figure 21. A Cluster of Three Item Outlines (Items 1-3) Created as Examples for Assessment Developers of the Types of Items to be Developed from the Framework

Item 1 Item 2 Item 3

Content: The nature of settlements is influenced Content: The nature of settlements is influenced Content: The nature of settlements is influenced by the interaction of characteristics of populations by the interaction of characteristics of populations by the interaction of characteristics of populations and physical environments. and physical environments. and physical environments.

Cognition: Identify geographic data that can help to Cognition: Find and describe spatial and temporal Cognition: Construct an explanation or prediction answer a question or solve a problem. patterns in data, or find data that match a pattern, for phenomena by comparing data to a model to help answer a question or solve a problem. or theory. Cognitive demand: strategic Cognitive demand: schematic Cognitive demand: schematic Item format: constructed response-short answer Item format: selected response Item format: constructed response Performance expectation: Identify resources and data that would help answer a question about Performance expectation: Describe patterns or Performance expectation: Make a prediction impact of policy decisions made for a settlement relationships shown between the maps using about how policy decisions made for a settlement on how the population interacts with the physical concepts of density and gradient. will impact the population and landscape using environment. observations about their relationship. Compare the two maps. Are there patterns to A city has a problem that people are not recycling where most bottles are found and the fewest bottles Justify whether a bottle deposit fee would be an plastic bottles, and the bottles are ending up in are found across the region? Where the bottles are, effective strategy for increasing recycling rates the rivers and lakes. The city officials are trying are there any patterns to where they appear within using the patterns observed in the maps. to determine if they should institute a deposit fee, the landscape? so they have asked your class to determine if a refundable bottle deposit would be an effective - Students should note where the highest density strategy for increasing recycling rates. Use the two of bottles occur relative to areas where deposit maps below to analyze how effective deposit fees fees are charged. have been in other regions. - Students should note where the highest density 1. One map shows the distribution of deposit fees of bottles occur relative to watershed features. for plastic bottles.

2. The other map shows the incidence of plastic bottles in watersheds (landfill). The map needs to make watersheds clear so that students can identify a relationship between rivers and density of bottles on the landscape.

Is there any other information you would need to perform this analysis?

The Road Map Project | Assessment Report | Chapter 5 | An Example Geography Assessment Framework 63 of 75 Geography Executive Context and Purposes of Geography Example Appendices Preface Assessment Recommendations References Summary Goals Assessment Assessment Framework A, B, C Framework

Conclusion

!e matrix, assessment speci#cations, project outline, AFGS21 can be used to align formative and summative shows how the framework matrix can be used to and items in this chapter comprise a speci#c assessment assessment goals, where formative assessments monitor ensure that assessment developers probe the full range framework. !e description of the process illustrates students’ progress toward the goals and summative of targeted objectives, and that assessments provide how a framework for a speci#c purpose can be created assessments evaluate how well students achieved those learners with a variety of opportunities to demonstrate from the AFGS21. In addition, this example shows how goals by the end of the instruction. Finally, the example mastery or reveal weaknesses for each objective.

The Road Map Project | Assessment Report | Chapter 5 | An Example Geography Assessment Framework 64 of 75 Geography Executive Context and Purposes of Geography Example Appendices Preface Assessment Recommendations References Summary Goals Assessment Assessment Framework A, B, C Framework

Chapter 6: Recommendations

Over the course of this report, we have built a case for intervals. Well-designed, properly timed assessments can Therefore, the Committee recommends: the role that assessment can play in improving teach- provide every stakeholder in the system, from students 2. Assessments for geography should reflect the ing and learning of geography. We also have presented to teachers to policy makers, with crucial information ways that geographic knowledge and skills are evidence that current assessments of geography, by for improving teaching and learning. used in the world, including using evidence-based reasoning, problem solving, and communication. and large, do not capture the information needed to Therefore, the Committee recommends: improve geography teaching and learning. 1. Current state and national efforts to evaluate the !e Committee has developed the 21st Century Assess- In this chapter, we present a set of speci#c recommenda- status of geographic literacy should be contin- ment Framework for the Geographical Sciences (AFGS21) tions for actions that will lead to the development of ued, and additional resources should be allocated to be used as a blueprint for this approach to assess- to improve and expand them. In particular, we assessments that will support instructional improvement ment, and we have provided guidelines for using recommend that the NAEP Geography assess- in geographical sciences. !ese recommendations are AFGS21 to create assessment frameworks for speci#c ment should be conducted at intervals consistent targeted at educators, policy makers, and funders. contexts and purposes. with other subjects of critical importance for workforce preparation and national security, such Therefore, the Committee recommends: A Focus on Assessment in Efforts to as science (which is conducted every four to five 3. The 21st Century Assessment Framework for Improve Geography Education years) and math (conducted every two years). the Geographical Sciences should be used as the basis for designing the next generation of !e Committee believes that a focus on assessment, A New Approach to the Assessment assessment frameworks and assessments for formative and summative, should be integral to any of Geography geography. e"orts to improve geographic education. According to our research, few classroom materials for geography are To be useful for instructional improvement, assess- Because large-scale tests in%uence assessments con- accompanied by high-quality assessments to assist teach- ments should capture the behaviors that the instruc- ducted at the district, school, and classroom levels, it is ers and students in monitoring learning. In addition, in tion is designed to cultivate. In the case of geography, particularly important that the tests represent the kind the 2010-2011 academic year, fewer than one-third of that includes the ability to answer questions and solve of 21st century geographic thinking on which we expect states required students to take a high school exit exam problems by reasoning with geographic evidence. Our students to be assessed. or end-of-course assessment that assessed geographic research indicates that existing geography assessments Therefore, the Committee recommends: knowledge or practices (Unpublished Geography Edu- do not re%ect the importance of these skills. As a result, 4. The NAEP assessment framework for geography cation National Implementation Project report, 2012). existing assessments evaluate performances that are not should be revised to reflect this 21st century !e National Assessment of Educational Progress representative of the objectives of geography education. approach to assessment of geographic thinking. (NAEP) assesses geography for elementary, middle, Further, current assessments do not signal to educators and high school students across the United States, but and learners what those objectives are in the way that !e Committee considers the version of AFGS21 in these tests have been occurring at seven- to nine-year assessments should. this report to serve as a starting point, not a #nal

The Road Map Project | Assessment Report | Chapter 6 | Recommendations 65 of 75 Geography Executive Context and Purposes of Geography Example Appendices Preface Assessment Recommendations References Summary Goals Assessment Assessment Framework A, B, C Framework

product. !is general assessment framework should improvement efforts. These should be addressed Therefore, the Committee recommends: to areas of high need and broad applicability. undergo continuing research and development to 8. The field should invest in training and For example: facility with map interpretation ensure that it can be used successfully to guide the professional development programs to cultivate and analysis at an elementary level; key issues development of assessments of the nature and quality the assessment professionals that the geography in human geography at elementary, middle, that we are advocating in this report. education community needs, and to prepare and high school levels; human-environmental teachers and policy makers to use and learn from Therefore, the Committee recommends: interactions at middle and high school levels; a new generation of assessments. 5. AFGS21 should be a focus of ongoing research environmental dynamics at elementary and and development in framework and assessment middle school levels; and facility with geospatial Knowledge Base development for geography that will lead to technologies at a high school level. continuous improvements in assessment of K–12 A lack of research has impeded the Committee’s e"orts geography knowledge and skills. Capacity Building to develop empirically based guidelines for determining 6. A program of research should be initiated what geographic content and practices students at !e degree to which the implementation of the immediately to study learning progressions in di"erent grade levels should be assessed on and how recommendations in this report is successful will geographic practices over the K–12 timeline and they should be assessed.12 to study techniques for assessing mastery of depend on the #eld’s capacity to execute them. Therefore, the Committee recommends: geographic practices at all levels. The results Speci#cally, we need professionals with the necessary of this research should be used to inform the geography expertise and assessment development 9. A substantial investment should be made in the refinement of the articulation of expectations for training and experience to develop e"ective assessment development of research and experience based content and cognition in AFGS21. frameworks, items, and instruments for geography. on: (1) the nature of understanding, reasoning, We need educators who have been trained in the best and learning geography to inform assessment Shared Frameworks and practices for incorporating assessments and their results design; and (2) techniques for assessing Assessments to inform teaching and learning in the classroom.11 understanding and reasoning in geography. Finally, we need educators and policy makers who have !e geography education reform community is small the expertise to make well-founded decisions based and has few resources. !erefore, these resources must 11 Recommendation 2 in the Road Map Project report on on the results of these assessments. Although expertise be applied strategically and e&ciently. Instructional Materials and Professional Development also relates with creating assessments and integrating them into to the preparation of teachers for the use of assessments (Schell, Roth, & Mohan, 2013). Therefore, the Committee recommends: classroom practice is not abundant in the broad 7. The field should develop a set of shared education community, it is vastly under-represented in 12 !e Road Map Project report on Geography Education frameworks and assessments to serve assessment the small and historically under-resourced geographic Research also contains recommendations about the need for research that can inform the design and use of assessments needs across a broad range of instructional education community. (Bednarz, He"ron, & Huynh, 2013).

The Road Map Project | Assessment Report | Chapter 6 | Recommendations 66 of 75 Geography Executive Context and Purposes of Geography Example Appendices Preface Assessment Recommendations References Summary Goals Assessment Assessment Framework A, B, C Framework

Appendix A: Additional Details About the Assessment Study

In this appendix, we provide additional details about the several textbooks, we chose one of the most widely used Table A1. Number of Items Coded from methods used in the study of current assessment practic- textbooks, and coded all of the items in each of the ran- Geography Resources by Grade Band es described in Chapter 3. Specifically, we describe how domly selected chapters. Large items were selected for study, how they were coded, and Classroom Total The methodological differences between sampling across Scale how the codes were analyzed. Elementary the large-scale and classroom assessment categories led us 109 18 127 school Item Selection to report the results separately. Selection of a representa- tive sample was far more straightforward for large-scale Middle school 131 214 345 The process of selecting items for analysis consisted of a assessments, so for the purpose of this report we focus the High school 172 49 221 results mainly on those items, though we show results of collection phase, a sampling phase, and a filtering phase. Total 412 281 693 classroom assessments where we find those results to be In the collection phase, we created a database of geo- particularly relevant. graphic education materials that were likely to have as- Table A2. Distribution of Resources Sampled for the Study sessment items, including all of the most recently released In the filtering phase, we eliminated items that did not assess geography by determining if the items assessed con- state, national, and international (English-speaking coun- Classroom Large Scale tries only) assessments. Our database also includes all tent or practices from any of the six essential elements in Resources Geography for Life: National Geography Standards (Heffron classroom geography units that were accessible through Number of resources 154 69 Social Studies School Service, Amazon.com, the National & Downs, 2012). Number of resources 88 29 Council for Geography Education, the College Board, The number of resources selected includes the number selected Social Studies Central, and the Library of Congress. In Number of resources selected through random sampling of the list of resources 69 12 many states, geography is not tested separately from other (Table A2). The number of resources coded eliminates coded subjects, so we included both the science and social stud- the resources with no geography items and also eliminates ies tests in those instances. the resources included in assessments that were not avail- able to the public. We randomly selected resources from the database sepa- rately by resource type, but we found that almost 60% of Item Coding Item Characteristics the classroom units selected had no assessments that fit our coding criteria, which resulted in under-sampling in Items selected for the study were coded according to The assessment setting describes whether the item is that category overall and a significant bias toward middle our taxonomy for geographic assessments (Table 6), designed as a proximal or distal evaluation of student school (Table A1). In addition, textbooks had so many which describes item characteristics, targeted abilities, learning. This category classifies items intended for a items that they would be overrepresented in the study if and confounding factors. In the sections below, we de- large-scale setting, including state, national, and inter- we sampled them using the same method, and there was scribe each of these coding categories and the codes that national tests, or in a classroom setting, which includes little variation between textbooks. So instead of sampling were used. items designed to accompany a geography unit or a ge-

The Road Map Project | Assessment Report | Appendix A | Additional Details About the Assessment Study 67 of 75 Geography Executive Context and Purposes of Geography Example Appendices Preface Assessment Recommendations References Summary Goals Assessment Assessment Framework A, B, C Framework

ography textbook. Program assessments are identified in Table A3. Essential Elements and Standards from Geography for Life (2nd ed.) the taxonomy as an additional setting, but items from The National Geography Standards: Geography for Life (2nd ed.) this setting were not included in this pilot study. 1. How to use maps and other geographic representations, geospatial technologies, and spatial thinking to understand and communicate information !e grade band indicates whether the item is targeted The World in 2. How to use mental maps to organize information about people, places, and environments to students in elementary, middle, or high school. In Spatial Terms in a spatial context rare cases items are intended for more than one grade 3. How to analyze the spatial organization of people, places, and environments on band, so we included those items with the lowest Earth’s surface targeted grade. 4. The physical and human characteristics of places Places and 5. That people create regions to interpret Earth’s complexity Item format records how the question was asked, Regions 6. How culture and experience influence people’s perceptions of places and regions speci#cally how structured the question and responses 7. The physical processes that shape the patterns of Earth’s surface are. Highly constrained response formats include Physical Systems multiple-choice items, which are characterized by 8. The characteristics and spatial distribution of ecosystems and biomes on Earth’s surface whether the question is written with a complete or 9. The characteristics, distribution, and migration of human populations on Earth’s surface incomplete stem (e.g., #ll-in-the-blank), in addition 10. The characteristics, distribution, and complexity of Earth’s cultural mosaics 11. The patterns and networks of economic interdependence on Earth’s surface to the less common true/false and matching formats. Human Systems Open-ended item structures include short answer, 12. The processes, patterns, and functions of human settlement which is limited to words and phrases or partial 13. How the forces of cooperation and conflict among people influence the division and control of Earth’s surface sentences, whereas the paragraph and essays category includes multiple-sentence responses. An additional 14. How human actions modify the physical environment Environment 15. How physical systems affect human systems open-ended item format that is occasionally used in and Society geography assessments asks students to construct a 16. The changes that occur in the meaning, use, distribution, and importance of resources

representation. Note: !e sixth element is not included because it involves geographic practices, which are described in the cognitive dimension.

Type of representation characterizes the representation(s) Geographic content describes the geographic content Geographic practices scored for the study are the used in the item so that we can collect data on what in- knowledge assessed by the item (Table A3). These geographic practices as described in AFGS21 (see formation, other than text, students are asked to evaluate categories are taken directly from the K–12 national Chapter 1), with the exception of collecting and as part of the item. For items that have a representation, standards recommendations Geography for Life. Each organizing geographic information, which was considered this category records if it is an illustration or photograph, of these categories is discussed in detail in Geography one practice category for this study, but has been split map or globe, graph or table, diagram, or document. for Life. The five categories included in the study are into two categories in AFGS21. These categories are Targeted Ability the essential elements that target geography content explained in depth in Chapter 4. If an item assesses a !ese categories describe what is being assessed by the knowledge. There is a sixth essential element, The Uses practice, but the practice is not specifically geographic, item, including the content, practices, and how students of Geography, which we consider to be outside of the such as reading a bar graph, the item was scored as are being asked to use the content and practices. content domain. requiring no geographic practices.

The Road Map Project | Assessment Report | Appendix A | Additional Details About the Assessment Study 68 of 75 Geography Executive Context and Purposes of Geography Example Appendices Preface Assessment Recommendations References Summary Goals Assessment Assessment Framework A, B, C Framework

The cognitive demands category characterizes the de- his or her ability. For example, an implausible answer well enough, the context being unfamiliar to most gree to which students are asked to process information choice can reduce the number of options, increasing the students, or not having enough supporting background as shown in Table A4 (Li, Ruiz-Primo, & Shavelson, probability that the student could choose the correct to make sense. 2006; Ruiz-Primo, Shavelson, Hamilton, & Klein, answer without using the targeted knowledge or skill Content accuracy describes problems with the sub- 2002). Items that ask students to respond with declara- (i.e., false positive). Similarly, poor wording might stance of the item. This category records whether items confuse a student, causing him to select an incorrect tive knowledge impose the lowest cognitive demand, have more than one answer choice that could be correct, answer choice even though he knew the correct answer while strategic knowledge requires the highest cogni- no answer choice is correct, there is a conceptual error, (i.e., false negative). tive level by asking students to understand a topic at a and if the representation is unnecessary for evaluating sophisticated level such that they know how and when Clarity includes descriptors of problems that students the item. to apply practices and content to a new problem in a might have with reading and interpreting the item. This Resistance to test-taking strategies is a category that new setting. category includes problems such as unclear questions describes ways that students can select or reject at least and/or answers, or representations that are unclear. Confounding Factors one answer choice using reasoning other than the tar- Some items also are written with an unnecessarily high geted knowledge or skill. This category includes criteria An item might be intended to assess a student’s ability reading load because it has too many words or it has that indicate if one answer choice is significantly longer across a given knowledge or skill domain, but if the words that are more technical or sophisticated than is or shorter than the others, or if some answer choices item is poorly written, we cannot be certain that the necessary. Items also are scored for insufficient informa- stand out because they seem more technical than the student’s performance on that item accurately re%ects tion, which includes representations not being labeled others. This category also includes criteria that indicate Table A4. The Categories of Cognitive Demands with Examples of Tasks That Typically Draw on These Demands if an answer choice seems unlikely to be true, or if there is a grammatical mismatch between the stem and at Cognitive Demands Tasks least one answer choice; both of these criteria make an • Identify, describe, classify answer choice less plausible. An additional criterion • Recognize a true statement Declarative records if an item has at least two answer choices that • Find relevant information in a text say or mean the same thing, allowing students to rule • Make a comparison between two variables out both using logic instead of the targeted geographic • Read a map skill or knowledge. Procedural • Collect information from a table, graph, map, or representation • Plot data on a table, graph, map, or representation Analysis of Coding Reliability • Make a comparison across multiple variables • Explain or predict a phenomenon using a general principle or model All items were coded across the geographic assessment Schematic • Perform evidence-based reasoning taxonomy by one researcher. A second researcher coded • Recognize and compare patterns 10% of the items to establish inter-rater agreement, and to test for consistency of coding between independent Strategic • Find a solution to a problem by devising an appropriate method of approach raters. Percentage agreement ranged from 71-93% (mean 83%), and the kappa coefficient ranged from

The Road Map Project | Assessment Report | Appendix A | Additional Details About the Assessment Study 69 of 75 Geography Executive Context and Purposes of Geography Example Appendices Preface Assessment Recommendations References Summary Goals Assessment Assessment Framework A, B, C Framework

showing fair to substantial agreement (0.249-0.654; amples of performances under each standard cover only and we felt that we did not have enough information p <0.001, 95% CI). Reported findings were calculated a small subset of the concepts and principles encom- about expectations for each standard to reliably code using the first rater’s results. passed by a standard. When faced with an item that re- the items. We coded according to our best estimation An area where inter-rater agreement was low was in quires students to have very context-specific factual recall of which standard best fit the knowledge needed, and coding for content category. Coders frequently found of geographic information, it was difficult to discern if when we could not make that determination, we coded it difficult to assign items to a specific standard in the we could reasonably expect that knowledge to be covered the item under the closest general standard heading. We Geography for Life standards. The standards are presented under any of the broadly defined standards. This dilem- scored items as “no content” only when the knowledge in the form of broad concepts and principles, and the ex- ma occurred frequently throughout the scoring process, needed could not be considered geographic.

The Road Map Project | Assessment Report | Appendix A | Additional Details About the Assessment Study 70 of 75 Geography Executive Context and Purposes of Geography Example Appendices Preface Assessment Recommendations References Summary Goals Assessment Assessment Framework A, B, C Framework

Appendix B: Example Assessment Framework

An excerpt from an example of an assessment framework for a high school unit on human settlement is shown below.

Download the full Example Assessment Framework at http://education.nationalgeographic.com/media/file/Example_Assessment_Framework_2.xlsx

The Road Map Project | Assessment Report | Appendix B | Example Assessment Framework 71 of 75 Geography Executive Context and Purposes of Geography Example Appendices Preface Assessment Recommendations References Summary Goals Assessment Assessment Framework A, B, C Framework

Appendix C: Assessment Committee Member and Staff Biographies

Daniel C. Edelson Statistical Reasoning for the Behavioral Sciences, Generalizability Elizabeth R. Hinde Theory: A Primer (with Noreen Webb), Scientific Research in Educa- Committee Chair tion (with Lisa Towne), and Assessing College Learning Responsibly: Committee Member Daniel C. Edelson is vice president for Education, National Accountability in a New Era. Elizabeth R. Hinde is associate professor and director of the Divi- Geographic Society, and executive director, National Geographic sion of Teacher Preparation at Arizona State University’s Mary Lou Education Foundation. As a curriculum designer, software devel- Jill Wertheim Fulton Teachers College. Prior to her career in higher education, oper, and educational researcher, Dr. Edelson has dedicated his Committee Research Director Dr. Hinde taught elementary school for 20 years. She is the author career to improving young people’s understanding of the world of numerous articles concerning social studies education and cur- they live in and their role in determining its future. In his position Jill Wertheim is program manager for evaluation and assessment riculum integration, and she has been recognized nationally for her as vice president for Education, he oversees National Geographic’s in National Geographic Society’s division of Education Programs. work in curriculum development. She was research director of the outreach to educators and its efforts to improve geographic and Dr. Wertheim has diverse experience in scientific research, assess- Arizona Geographic Alliance’s GeoLiteracy and GeoLiteracy for geoscience education in the United States and abroad. This work ment framework development, and assessment research and devel- English Language Learners programs, and she was a member of the includes the creation of educational materials for learners of all opment. Her experience includes working for four years at AAAS curriculum development team of iCivics.org. In 2005, Dr. Hinde ages, the delivery of professional development for educators, the Project 2061 as a research associate performing research on learning received the National Council for Geographic Education’s Distin- implementation of public engagement programs, advocacy on around earth science topics, research on assessment in earth science, guished Teaching Award, and she also received the 2010 Geography behalf of geographic education in policy discussions, and grant- as well as developing classroom assessment items and instructional Excellence in Media Award. She is past-president of the Arizona making to support geographic literacy initiatives throughout the materials, and evaluating items, instructional materials, and stan- Council for the Social Studies, teacher consultant with the Arizona United States and Canada. He has written extensively on motiva- dards. Her scientific research includes studies on using fossils Geographic Alliance, past-chair of the National Council for the tion, classroom teaching and learning, educational technology, to uncover evolutionary patterns in South American mammals. Social Studies Steering Committee, and is active in numerous state and teacher professional development. He is an author of more and national professional organizations. than 50 papers in journals, edited books, and conference proceed- Barbara Hildebrant ings, including The Cambridge Handbook of the Learning Sciences, The International Handbook on Science Education, Journal of the Committee Member Marianne Kenney Learning Sciences, Journal of Research on Science Teaching, and Barbara Hildebrant is senior director, College Board Programs, Committee Member The Science Teacher. Educational Testing Service (ETS). She brings more than 13 Marianne Kenney is a social studies specialist/evaluator at Denver years of experience in assessment development to the Road Map Richard J. Shavelson Public Schools. Ms. Kenney serves in the Teacher Effectiveness Unit Project. Her professional expertise includes academic program funded by a grant from the Bill and Melinda Gates Foundation. Committee Co-Chair review, higher education, and testing program management. Her She has been a senior consultant at McREL, the region’s federally Richard J. Shavelson is chief scientist and partner at SK Partners, relevant experience includes developing and reviewing curriculum funded research laboratory, and the state social studies content LLC, and the Emeritus Margaret Jacks Professor of Education and framework and form assembly specifications, developing tests, and specialist for the Colorado Department of Education. She has been Professor of Psychology (by courtesy), and former I. James Quillen managing scoring activities. Her academic areas of expertise are an- directly involved in the development of standards at the national Dean of the School of Education at Stanford University, and senior thropology, geography, and world history. Prior to joining ETS, she and state levels. She has directed many summer institutes to assist fellow in the Woods Institute for the Environment. He served as taught geography and anthropology classes at Rutgers University. districts in designing both district and classroom performance president of the American Educational Research Association, and In 2008, she received the Association of American Geographers assessments, as well as units of instruction that align with national, he is a fellow of four professional associations and a Humboldt (AAG) Gilbert Grosvenor Honors for Geographic Education in state, and/or district standards. Ms. Kenney has been on the advi- Fellow (Germany). His work includes assessment of undergradu- recognition of her exceptional record of leadership in advancing sory committee for the 1988, 1994, and 2001 National Assessment ates’ learning, including the Collegiate Learning Assessment, geography education both at the university level and in of Educational Progress Geography tests, and she has worked with accountability in higher education, assessment of science achieve- K–12 schools. She earned her PhD in Geography from Rutgers the Council of Chief State School Officers to collaboratively de- ment, the enhancement of minorities’ performance in organic University, her MA in Anthropology from the University velop state test items. She is an experienced presenter at workshops chemistry, and the role of mental models of climate change on of Washington at Seattle, and her BA in Anthropology from and conferences and has published numerous articles for education sustainability decisions and behavior. His publications include Drew University. journals. Ms. Kenney taught high school geography for 17 years.

The Road Map Project | Assessment Report | Appendix C | Assessment Committee Member and Staff Biographies 72 of 75 Geography Executive Context and Purposes of Geography Example Appendices Preface Assessment Recommendations References Summary Goals Assessment Assessment Framework A, B, C Framework

Bob Kolvoord Jody Smothers Marcello Education (NCGE). In 2007 he was awarded the NCGE George Miller Award for geography education contributions, and in 2012 he Committee Member Committee Member received the Association of American Geographers Gilbert Grosvenor Bob Kolvoord is interim dean, College of Integrated Science and Jody Smothers Marcello is a teacher at Sitka High School, Sitka, Honors for Geographic Education. Engineering, James Madison University. Dr. Kolvoord also serves as Alaska. Dr. Smothers Marcello is a National Board Certified teacher the interim head of Engineering and as a professor of Integrated Sci- and educator; she teaches AP Human Geography (APHG), global Maria Ruiz-Primo ence and Technology. His research interests focus on the use of data issues, history, and English. She has received multiple teaching Committee Member visualization and geospatial technologies in the K–16 classroom and awards at the state and national levels, including from the National Maria Ruiz-Primo is an associate professor at the School of Education the development of student spatial thinking skills through the use Council for Geographic Education (NCGE) and National Council for the Social Studies (NCSS). She has served as NCGE president, and Human Development, University of Colorado Denver. Her of these technologies. He is a co-creator of the Geospatial Semester on the NCSS Board, on the National Board for Professional Teach- work focuses on two strands: assessment of students learning at both program (http://www.isat.jmu.edu/geospatialsemester), and with ing Standards Social Studies-History standards committee, and as a large-scale and classroom level, and the study of teachers’ assessment Kathryn Keranen is the co-author of the award-winning Making Spa- table leader for the APHG College Board reading. She has authored practices. Her publications reflect these two strands: developing and tial Decisions Using GIS (1st and 2nd editions) and Making Spatial multiple articles and curricula, including recent publications for the evaluating different strategies to assess students’ learning, such as Decisions Using Remote Sensing (forthcoming). He was the recipient of College Board and NCGE. She was a teacher contributor to the 2011 concept maps and students’ science notebooks, and studying teachers a 2010 Commonwealth of Virginia Outstanding Faculty Award for book, The American Public School Teacher: Past, Present, and Future, informal and formal formative assessment practices, such as the use teaching with technology. she is a contributor to the Association of American Geographers’ of assessment conversations and embedded assessments. Her recent Center for Global Geography Education. She edits The Geography work focuses on the development and evaluation of assessments that David A. Lanegran Teacher journal. She received BS and MEd degrees from Texas A&M are instructionally sensitive and assessment instruments of formative assessment practices in the classroom. Committee Member University. David A. Lanegran is John S. Holl Professor of Geography, Robert W. Morrill Peter Seixas Macalester College, St. Paul Minnesota. Dr. Lanegran is an urban Committee Member Committee Member geographer specializing in urban issues related to city planning, urban development, and historic preservation. He has conducted extensive Robert W. Morrill is professor emeritus, Geography Virginia Tech, Peter Seixas is professor and Canada Research Chair in the Depart- comparative research on urban planning around the world and has 1973-2003. Dr. Morrill serves as co-coordinator, Virginia Geo- ment of Curriculum and Pedagogy at the University of British Columbia. Dr. Seixas is director of the Centre for the Study of His- published and spoken widely on urban, agricultural, and cultural ge- graphic Alliance. He has received human geography and geography torical Consciousness, and a member of the Royal Society of Canada. ography. He has been a driving force in Macalester’s nationally recog- education grants from the National Science Foundation, National Geographic Society, Commonwealth of Virginia, and Foundation He taught high school social studies in Vancouver over the course of nized programs to improve the teaching in geography in elementary for the Improvement of Post Secondary Education. Dr. Morrill led 15 years and earned a PhD in history from the University of Califor- and secondary schools. He is coordinator of the Minnesota Alliance study abroad programs and teacher institutes to Switzerland, Italy, nia at Los Angeles. He is the author of numerous articles on history for Geographic Education, and the chief reader of the Advanced Germany, The Netherlands, Canada, Ecuador, Brazil, and New education in Canadian and international journals, editor Theorizing Placement Human Geography exam. He has directed 70 summer Zealand. He was a Fulbright Research Fellow (1986), University of Historical Consciousness (University of Toronto Press, 2004), and co- institutes for geography teachers. He has received several honors, Turku, Finland. A primary author for Guidelines for Geographic Edu- editor, with Peter Stearns and Sam Wineburg, of Knowing, Teaching including Macalester’s Jefferson Award for Teaching Excellence; The cation (1984) and Geography for Life: National Standards in Geography and Learning History: National and International Perspectives (NYU George J. Miller Award, from the National Council for Geographic (1994), Writing Committee member for the Geography Framework Press, 2000). He is director of the pan-Canadian Historical Thinking Education; and the Gilbert Grosvenor Honors for Geographic Edu- for the National Assessment for Educational Progress. Dr. Morrill’s work Project (www.historicalthinking.ca), which aims to promote critical cation from the Association of American Geographers. His publica- has been published in geography and education journals, curriculum historical literacy through provincial history curricula, textbooks, tions include 12 books and more than 60 articles and chapters in monographs, atlases, and national geography education reports. In assessments, and professional development. His current research aims professional publications. 1989, he served as president of the National Council for Geographic toward the development of robust assessments of historical thinking.

The Road Map Project | Assessment Report | Appendix C | Assessment Committee Member and Staff Biographies 73 of 75 Geography Executive Context and Purposes of Geography Example Appendices Preface Assessment Recommendations References Summary Goals Assessment Assessment Framework A, B, C Framework

References

Abler, R. F. (1987). What shall we say? To whom College Board. (2009). Science college board standards Heffron, S. G., & Downs, R. M. (Eds.). (2012). shall we speak? Annals of the Association of American for college success™. Science. New York, NY: Author. Geography for life: National geography standards Geographers, 77(4), 511–524. (2nd ed.). Washington, DC: National Council for Fu, A. C., Raizen, S. A., & Shavelson, R. J. (2009). Geographic Education. Association of American Geographers. (1966). High The nation’s report card: A vision of large-scale science School Geography Project: Geography in an Urban Age. assessment. Science, 326, 1637–1638. Helburn, N. (1998). The High School Geography New York, NY: Macmillan. Project: A retrospective view, Social Studies, 89. Geography Education National Implementation Project. Baerwald, T. J. (2010). Prospects for geography as an (1987). K-6 geography: Themes, key ideas and learning Joint Committee on Geographic Education. (1984). interdisciplinary discipline. Annals of the Association of opportunities. Washington, DC. Guidelines for Geographic Education. Washington DC: American Geographers, 100(3), 493–501. Association of American Geographers and National Geography Education National Implementation Project. Council for Geographic Education. Bednarz, S. W., Heffron, S., & Huynh, N. T. (Eds.). (1989). Geography in grades 7-12: Themes, key ideas and (2013). A road map for 21st century geography education: Li, M., Ruiz-Primo, M. A., & Shavelson, R. J. (2006). learning opportunities. Washington, DC. Geography education research (A report from the Geogra- Towards a science achievement framework: The case phy Education Research Committee of the Road Map Geography Education National Implementation of TIMSS 1999. In S. J. Howie & T. Plomp (Eds.), for 21st Century Geography Education Project). Wash- Project. (2012). State of Geography Education in the Contexts of learning mathematics and science: Lessons ington, DC: Association of American Geographers. United States. Unpublished report. learned from TIMSS (pp. 291–311). New York, NY: Retrieved from www.natgeoed.org/roadmap Routledge. Geography Education Standards Project (1994). Black, P., & Wiliam, D. (1998). Assessment and class- Geography for life: National geography standards 1994. National Assessment Governing Board. (2008). Science framework for the 2009 National Assessment room learning. Assessment in Education: Principles, Policy Washington, DC: National Geographic Research and of Educational Progress (NAEP). Retrieved from & Practice, 5(1), 7–74. Exploration. http://www.nagb.org/publications/frameworks.html Black, P., and D. Wiliam. (2004a). Classroom assess- Goals 2000: Educate America Act. (1994). Public Law ment is not (necessarily) formative assessment (and National Assessment Governing Board. (2010). No. 103–227. vice-versa). Yearbook of the National Society for the Study Geography framework for the 2010 National Assessment of Education, 103(2): 183–188. Hanson, Susan. (2004). Who are ”we”? An important of Educational Progress (NAEP). Retrieved from question for geography’s future. Annals of the Association http://www.nagb.org/publications/frameworks.html Black, P., and D. Wiliam. (2004b). The formative pur- of American Geographers, 94(4): 715–722. pose: Assessment must first promote learning. Yearbook National Center for Education Statistics. (2011). The of the National Society for the Study of Education, 103(2): nation’s report card: Geography 2010. Washington, DC: 20–50. Institute of Education Sciences, U.S. Department of Education.

The Road Map Project | Assessment Report | References 74 of 75 Geography Executive Context and Purposes of Geography Example Appendices Preface Assessment Recommendations References Summary Goals Assessment Assessment Framework A, B, C Framework

National Center for Education Statistics. (2012). National Research Council. (2011). Assessing 21st Development Committee of the Road Map for 21st Science in action: Hands-on and interactive computer century skills. Washington, DC: The National Century Geography Education Project). Washington, tasks from the 2009 Science Assessment. Retrieved Academies Press. DC: National Council for Geographic Education. from http://nces.ed.gov/nationsreportcard/pubs/ Retrieved from www.natgeoed.org/roadmap National Research Council. (2012). A framework for main2009/2012468.asp K–12 science education: Practices, crosscutting concepts, Shavelson, R. J., Young, D. B., Ayala, C. C., Brandon, National Commission on Excellence in Education. and core ideas. Committee on a Conceptual Framework P. R., Furtak, E. M., Ruiz-Primo, M. A., Tomita, M. (1983). A nation at risk. The imperative for education for New K–12 Science Education Standards. Board K., & Yin, Y. (2008). On the impact of curriculum- reform. Washington, DC: U.S. Government Printing on Science Education, Division of Behavioral and embedded formative assessment on learning: A Office. Social Sciences and Education. Washington, DC: The collaboration between curriculum and assessment National Academies Press. developers. Applied Measurement in Education, 21(4), National Geographic Society. (2011). Geo-literacy 295–314. coalition responds to 2010 National Assessment of No Child Left Behind Act of 2001, Pub. L. No. Geography Education with call to action [Press release]. 107–110, 115 Stat. 1425 (2002). Taaffe, E. J. (1974). The spatial view in context. Annals of the Association of American Geographers, 64, (1): 1–16. Retrieved from http://press.nationalgeographic. Pattison, W. D. (1964). The four traditions of com/2011/07/19/geo-literacy-coalition-responds-to- geography. Journal of Geography, 63, no. 5, 211–216. U.S. House of Representatives. (2009). Departments of 2010-national-assessment-of-geography-education- Transportation and Housing and Urban Development with-call-to-action/ Quellmalz, E. S., Griffin, M., Hurst, K., Kreikemeier, and related agencies appropriations Act, 2010. P., Rosenquist, A., & Zalles, D. (2004, April). Conference report to accompany H.R. 3288. Retrieved National Research Council. (1997). Rediscovering Integrated performance assessments with technology from www.gpo.gov/fdsys/pkg/CRPT-111hrpt366/pdf/ Geography: New Relevance for Science and Society. (IPAT): Design model and prototypes. Presented at the CRPT-111hrpt366.pdf Washington, DC: The National Academies Press. annual meeting of the American Educational Research Association, San Diego, CA. Virginia Department of Education. (2008). History and National Research Council. (2001). Knowing what social studies standards of learning for Virginia Public students know: The science and design of educational Ruiz-Primo, M. A., Shavelson, R. J., Hamilton, L., & Schools. Retrieved from http://www.doe.virginia.gov/ assessment (J. W. Pellegrino, N. Chudowsky, & Klein, S. (2002). On the evaluation of systemic science testing/sol/standards_docs/history_socialscience/index. R. Glaser, Eds.). Washington, DC: The National education reform: Searching for instructional sensitivity. shtml Academies Press. Journal of Research in Science Teaching, 39(5): 369–393. Vygotsky, L. S. (1978). Mind and society: The National Research Council. (2010). Understanding Schell, E. M., Roth, K. J., & Mohan, A. (Eds.). development of higher mental processes. Cambridge, MA: the changing planet: strategic directions for the (2013). A road map for 21st century geography education: Harvard University Press. geographical sciences. Washington, DC: The National Instructional materials and professional development (A Academies Press. report from the Instructional Materials and Professional Wiggins, G. P., & McTighe, J. (2005). Understanding by design. Association for Supervision and Curriculum Development.

The Road Map Project | Assessment Report | References 75 of 75