Competencies That Count: Strategies for Assessing High-Performance Skills

LAB Working Paper No. 2 ompetencies C That Count Strategies for assessing high-performance skills

Lili Allen

his guide provides a “road map” to the various ways that schools Northeast and Islands and employers assess high-performance competencies—such as Regional Educational T problem solving, information management, and communication Laboratory and negotiation skills. The importance of such competencies is becoming increasingly evident because of their relevance both to LAB school and to job performance. The guide begins with a brief a program of analysis of why it is critical to assess these skills in light of both the The Education Alliance current standards environment in education and key trends in the at Brown University economy. The guide includes an overview of tools and materials for assessing high-performance competencies and profiles of selected 222 Richmond Street, Suite 300 tools and materials. Providence, RI 02903-4226 The methodologies described here range from “off-the-shelf,” short- Phone: 401/274-9548 answer tools to relatively expensive, hands-on assessments. While Fax: 401/421-7650 some assess high-performance competencies in a relatively tradi- E-mail: [email protected] tional manner, through multiple-choice questions about hypotheti- Web: www.lab.brown.edu cal scenarios, others document and assess the ability of students to grapple with messy, real-world problems and to choose the appropri- ate tools and methodologies for accomplishing a task. The former assess knowledge of cross-cutting competencies and are likely to withstand challenges of reliability and validity, while the latter assess how students use cross-cutting competencies in practice and demon- in partnership with strate what students can do in actual situations. JOBS FOR Jobs For the Future THE FUTURE Because of the variability in methodologies, the all students to the same high standards, but, at the profiles range from simple descriptions of existing same time, they threaten to crowd everything except materials to extended narratives of program or system traditional academics out of the curriculum. development. Several of these strategies—especially This state of affairs is surprising given the foundation most of those in the section “Competency Assessment in Schools”—outline processes that schools and school upon which the standards movement rests. Early in the standards movement, there was much talk about systems have developed in collaboration with partners. The New Hampshire Competency-Based Transcript, structural changes in the nature and organization of work, the labor market, and the global economy, and for example, is the story of a forward-thinking state department of education that engaged with business how these changes create a need for a new set of high- performance competencies. Educators, policymakers, and higher education partners to create a statewide system to capture student learning of high-perfor- industry leaders, and academics concurred: to succeed in today’s economy, adults need cognitive, interactive, mance competencies. Another state could adapt the resulting transcript, but, more importantly, New and problem-solving skills, along with more sophisti- cated academic skills than school systems were offer- Hampshire’s system-building process could be used to develop a tool that reflects another state’s needs and ing. Most notably, a 1991 Labor Department blue- ribbon task force identified so-called SCANS skills strengths in educational and workforce development. that it considered critical for economic success in the Other strategies have been developed for a specific modern workplace.1 According to the task force, setting (e.g., community colleges, adult education these skills included cognitive skills, personal qualities, centers) and could be adapted for alternative settings. and competencies such as the abilities to collect and For example, teachers seeking ways to assess student manage information, identify and use resources, work skill gains in the completion of projects could adapt with others, and understand and design systems. At the technique of behavioral interviewing, which is the same time, parents in upper-income communities widely used in the corporate sector. We believe that became concerned that their children might not be the methodologies described here can promote cross- gaining the critical thinking skills that are essential for fertilization among the secondary education, adult professional careers, while literature emerging from education, corporate, and community college fields graduate schools stressed the need for students who and inspire educators to try a range of strategies to are completing advanced academic work to have such teach and assess these high-performance competencies. high-performance skills as teamwork and problem- solving.

In recent years, however, the notion of a common core Why Assess High-Performance of cross-cutting skills has been all but forgotten in the Competencies 1 As we turn the corner to the new millennium, more SCANS stands for Secretary (of Labor’s) Commission on Achieving Necessary Skills. See U.S. Department of Labor, and more states and districts are implementing new Secretary’s Commission on Achieving Necessary Skills, high-stakes exit exams that hold both a promise and a What Work Requires of Schools: A SCANS Report for America 2000 (Washington, DC: U.S. Department of threat for young people: these exams promise to hold Labor, 1991).

2 Competencies That Count: Strategies for assessing high-performance skills flurry of high-stakes, state- and district-level, stan- They are adding the assessment of competency assess- dardized testing. For reasons of simplicity, cost, and ments—the assessment of high-performance or cross- validity, most states have introduced standardized cutting skills—to their portfolio of measurement tools. tests that reward the mastery of traditional subject Competency assessments are attractive for a number of matter and thereby drive teachers to focus their classes on that content. Because a student’s achieve- reasons: ment of a certain level of academic skill and knowl- Competency assessments can provide a means to edge traditionally represents a gateway to college, capture learning that takes place in workplace and most educators and the public also perceive these as community-based learning experiences. the best predictors of a person’s ability to thrive in They can indicate the extent to which students the economy and as a community member. who may not perform well on traditional stan- Nevertheless, traditional academic skills and knowl- dardized tests have gained important knowledge edge are not enough to help students address the and skills that will affect their future success. ambiguous, complex, semi- or unstructured prob- They can indicate ways to enrich curricula to lems of the real world. In fact, while often seen as include additional cross-functional skills. “career competencies” of value only in the workplace, high-performance skills are as important for success They can provide a bridge to local workforce development initiatives. in K–12 and postsecondary education as they are in employment. Their centrality increases the higher Perhaps most importantly, high performance assess- one goes in education, and, as a result, they are ments can show us what students can do rather than becoming part of curricula and pedagogy in law focusing only on where their deficits lie. schools, medical schools, and business schools. Instead of artificially separating career competencies from academic or intellectual competencies, we need to focus on preparing young people to succeed in all Tools and Methodologies for levels of formal education, in the workplace, and in Assessing High-Performance community life. Competencies

As any educator knows, the best way to encourage We have cast a wide net in surveying and identifying the teaching and learning of a more integrated set of methodologies for assessing cross-cutting competencies. skills is to assess those skills. However, many schools, While some originated in the high school building, districts, and communities choose not to try to others were designed to help corporations identify incorporate large-scale performance assessments into qualified entry-level employees or were created by their content standards and assessment systems, in collaborations among community-based organizations, part due to complications concerning validity. At the schools, and business partners. This list is by no means same time, and even though high-stakes, standard- exhaustive. Instead, it identifies types of assessments, ized tests are the most visible aspect of standards- presents one or two examples of each type, and, to the based reform, some communities are choosing to use extent possible, illustrates how the assessments have multiple measures to demonstrate student outcomes. been used.

Competencies That Count: Strategies for assessing high-performance skills 3 For ease of navigation around the methodologies, the nity to learn. Alternatively, some schools may assessments are grouped here into five categories: want to create a process to identify, with their students or faculty, what skills they would like On-demand tests, both multiple-choice and short- students to attain through projects, courses, or answer: Work Keys (ACT, Inc.); Workplace course sequences. For example, as a method of Success Skills (Learning Resources, Inc.) developing faculty support for reform, schools involved in the Coalition of Essential Schools Self-assessments: Working (H & H Publishing engage faculty in determining what students Company) should be able to know and do. Similarly, an Competency assessment in schools: New Standards individual teacher might work with a class to Project (National Center on Education and the determine what skills should be gained through Economy and the University of Pittsburgh); New particular assignments or activities, or a commu- Hampshire’s Competency-Based Transcript; nity partnership might bring together employers Coalition of Essential Schools/Transitions and school partners to identify skills that work- Project; SCANS 2000 Center/AES International based learning experiences should offer. 2 Competency assessment in the workplace: Massa- Determine the primary purpose of the assessment. chusetts Work-Based Learning Plan; Behavioral It may be: Interviewing; SHL Job Simulations To give feedback to the student for mid- Competency assessment in adult education settings: course corrections as part of the learning Equipped for the Future process. The Massachusetts Work-Based Learning Plan has mid-course check-ins embedded in the course of documentation.

To offer additional skills not gained elsewhere How to Approach the Assessment in the curriculum. For example, through an Tools adaptation of the behavioral interviewing process, students could become more reflec- As you review the descriptions of tools and method- tive of their learning and learn to gather ologies, bear in mind the context in which you plan to evidence of and articulate the skills they have implement this type of assessment. Adapting the tools gained. and methodologies requires various levels of involve- To provide a culminating evaluation. Tradi- ment of and preparation by the teacher and commu- tionally, assessments have been used for this nity partners. In general, to make it possible to assess purpose. cross-cutting competencies, educators and business/ community partners need to: To document, and perhaps accredit, skills so that students can convey their skills to 1 Identify the specific skills that a project or com- postsecondary institutions or employers. Most munity experience, such as an internship, would of the “ready-to-use” evaluations are legally be likely to develop. Most of the methodologies defensible. The New Hampshire competency- described here identify and define what skills based transcript was designed collaboratively they assess; schools might wish to use a tool that to be a portable record of learning for both aligns with what their students have an opportu- postsecondary institutions and employers.

4 Competencies That Count: Strategies for assessing high-performance skills Tools and Methodologies for Assessing High Performance Skills

Tool/method Level of system Audience/ requires change required/ Assessment Method Uses Competencies customization assumed Work Keys Paper-and-pencil; Schools, regional Applied No None some video/audio partnerships, mathematics; businesses, states applied technology; listening; locating information; observation; reading for information; teamwork; writing Workplace Video-based paper- Schools, regional Interacting with No None Success Skills and-pencil partnerships, adult others; listening; workforce structuring work development activities; programs, trainability; graphs businesses and charts Skill Command Rubric Schools, regional Teamwork; No None partnerships, interpretation and businesses use of information; problem-solving SHL Job Simulations Job interviews in Functioning in With adaptation, None Simulations private sector group and team could be used by settings; organizing schools and prioritizing tasks; conducting research, analysis, and idea generation to solve problems Working Paper-and-pencil Schools, colleges, Taking responsibility; No None self-assessment adult workforce working in teams; development persisting; sense of programs quality; lifelong learning; adapting to change; permanent problem-solving; information processing; systems thinking New Standards Work samples with Districts, for Problem solving; No. Some districts Requires professional Applied Learning rubrics implementation in communication customize by development to help Standards classrooms information; developing locally teachers incorporate learning and self- based exemplary standards into management; work samples teaching practice working with others and to assess student performance New Hampshire’s Exemplars/ Piloting in fourteen Decision making Yes. Districts in System is the result Competency- locally developed schools in New and problem-solving; other states would of collaboration Based Transcript rubrics and Hampshire; eventual self-management; have to customize between state DOE, System portfolios use by all New communication; because the post-secondary, and Hampshire schools ability to work with transcript is based business partners; others; information/ on New Hampshire’s requires professional use of technology, curriculum development for research, analysis frameworks implementation Coalition of Rubrics/portfolios Piloting in two Coalition of Yes. Individual Requires agreement Essential Schools/ as part of schools in California Essential Schools’ schools identify own with postsecondary Transitions Project alternative transcript Habits of Mind, learner outcomes institutions to credit e.g., habits of and revise rubrics transcript for inquiry; technology; accordingly admissions; requires collection and professional organization of development on information; scoring for teachers communication of ideas; effective cooperation with others; lifelong learning

Competencies That Count: Strategies for assessing high-performance skills 5 To provide an opportunity for a teacher, On-Demand Tests, Both Multiple- school, district, or community to determine whether the curriculum overall is delivering Choice and Short-Answer particular skills. Work Keys has been used in Several companies have developed “ready-to-use” this regard in several localities. assessment systems that businesses can use to assess job To give a diagnostic report to a community applicants or that schools or industry/education regarding the skills students are gaining partnerships can use to assess student and worker skill through the existing curriculum. For example, levels. These methodologies have the advantage of Ohio used Work Keys to determine whether being legally defensible. Also schools, community students were prepared for the state’s fastest- organizations, and school/community partnerships can growing jobs, then devised a strategy to adapt them to identify skill gaps and the training needs address skill gaps. of students and potential workers. These assessment 3 Determine what “evidence” of student learning systems identify knowledge of competencies, rather and achievement the relevant stakeholders will than the ability to apply competencies in context. require. Most often, these stakeholders are “gatekeepers” such as employers and institutions WORK KEYS of higher education. The New Hampshire Department of Education and the California Work Keys is a three-stage employability skills assess- Transitions Project have focused on making ment tool that includes job analysis, assessment, and credible the attainment of cross-cutting compe- instructional support. Widely used by industry, educa- tencies to postsecondary institutions and employ- tion systems, and business/education consortiums, the ers. The SCANS 2000 Center’s Career Transcript Work Keys system is built around a set of scales that system allows users to post results from Work measure both the skills of individuals and the compe- Keys and from Work Success Skills for a portable tency levels required for successful job performance. “passport” of learning. Work Keys is a program of ACT, Inc., an organization 4 Determine how much change your system—school most widely known for its college admissions testing department, school/community partnership, or service. school—will be willing to undertake. For example, implementation of the New Standards Work Keys provides highly detailed information about requires professional development to help skill levels in eight “foundation skills,” or “the skills teachers incorporate the standards into teaching needed to learn other skills:” practice. Several of the tools described here require the creation of “assessor networks” within Applied Mathematics: Applying mathematical or across schools so that teachers and other reasoning to work-related problems; assessors can come to agreement on process and Applied Technology: Basic principles of mechanics, standards. In addition, many of the tools identi- electricity, fluid dynamics, and thermodynamics as fied here require adaptation by local users, as they apply to machines and equipment found in noted in matrix above. the workplace, with the emphasis on problem- solving;

6 Competencies That Count: Strategies for assessing high-performance skills Listening: Listening to and understanding work- the state’s fast-growing jobs and assess the skill levels related messages; of 14,000 high school seniors from urban, suburban, and rural communities. The results indicated that only Locating Information: Locating, comparing, summarizing, and making decisions using one in fourteen seniors met the profile across four skill information in workplace graphics (e.g., dia- areas (applied mathematics, reading for information, grams, floor plans, tables, forms, graphs, charts, applied technology, and locating information). The instrument gauges); initiative’s recommendations included raising academic standards, increasing opportunities for work- Observation: Noticing details and paying atten- based learning, and continuing the use of skill-based tion to instructions and demonstrations; assessments. Reading for Information: Reading and understanding work-related instructions and policies in In California, the Tri-Valley Education Collaborative memos, bulletins, notices, letters, policy manuals, has used Work Keys as a diagnostic tool to pinpoint and governmental regulations; needed changes in curricula and, less extensively, with a small group of students to assist them in undertak- Teamwork: Choosing behaviors and/or actions ing more careful career planning. In 1998, the Col- that simultaneously support relationships within laborative piloted the Work Keys assessment with the team and lead toward the accomplishment of about 500 students from comprehensive and alterna- work tasks; and tive schools, then gathered district leaders to discuss Writing: Writing work-related messages and aggregated results for their schools. The Collaborative summaries. hopes to build on that pilot by engaging schools in extended discussions about what curricular changes Each skill area is scaled from the lowest level for which would be necessary to improve student scores. employers are willing to test up to the point at which specialized training is needed. Several of the skills are Work Keys, ACT Inc, 3355 Lenox Road, NE, Suite 320, assessed through traditional, paper and pencil testing, Atlanta, GA 30326-1332 (404) 231-1952; while others incorporate the use of audio or videotapes www.act.org/workkeys/tour/index.html that provide prompts or scenarios to which the student must respond. Testing time ranges from 40 to 80 minutes. WORKPLACE SUCCESS SKILLS Work Keys materials suggest that students work with counselors to take the assessments and pinpoint areas Learning Resources, Inc., provides a video-based, for improvement. The accompanying Targets for criterion-referenced SCANS assessment used in high Instruction manuals are designed to be incorporated schools, businesses, postsecondary institutions, and into classroom curricula; they include descriptions of welfare training programs. The Workplace Success the skill levels, examples of skill building strategies, Skills (WSS) assessment system was developed and and work-related scenarios at each skill level. validated in collaboration with industry representatives. Educators use it to assess curricula and to assist In Ohio, the Ohio Business Roundtable and the Ohio students in identifying strengths and weaknesses; Department of Education hired ACT to both profile employers use it to screen applicants for entry-level

Competencies That Count: Strategies for assessing high-performance skills 7 jobs and to give feedback to employees so that they If all five modules are used, the assessment takes about may upgrade their work-relevant skills. WSS is also 2.5 hours and requires a PC with either Internet access being used by the SCANS 2000 Center, described or a modem, a VCR with monitor, and paper and a below, in three ways: (1) as a pre- and post-test to pencil. The participant views the video scenarios and assess community-college students’ acquisition of puts answers to questions onto an answer sheet. A test SCANS skills; (2) to assess the basic skills of partici- administrator then transfers the answers to the PC, pants in welfare-to-work programs for the purpose of which sends them to a host computer for scoring. providing developmental feedback; and (3) as part of Finally, the host computer transmits a comprehensive an incumbent worker/organized labor program funded feedback report back to the local computer. Because by the U.S. Department of Labor. the host computer does the scoring, no professional development costs are associated with WSS. As of WSS assesses thinking skills (creative thinking, deci- September 2000, a modem will no longer be required: sion making, problem solving, knowing how to learn, data and reports may all be scored by the Internet. and reasoning) and personal qualities (responsibility, sociability, self-management, and commitment to The feedback report contains: quality). Probability-of-success information, based on a There are five sections to the assessment: comparison of the examinee’s scores against a validation group (the database of scores exceeds Interacting with Others: The participant views one million people); brief vignettes of work situations with hourly workers and their supervisors as central charac- A rank ordering of the participant’s facilitation ters. Periodically, the video stops; the participant skills, influence skills (guiding the actions and must respond to multiple-choice questions thinking of others), customer service orientation, requiring judgment about actions or decisions. problem-solving skills, and listening skills;

Listening: The participant views video vignettes A detailed report showing areas where the that present information and instructions orally; participant’s performance was satisfactory and they must answer multiple-choice questions where development is warranted; and about factual content interpretation. Suggested activities for improving performance Structuring Work Activities: A supervisor gives the in the assessed skill areas. participant oral instructions on a complicated In addition to the feedback report, resources designed task; the participant must chronologically list the actions required to complete the task. to support educators and trainers include a facilitator’s guide to manage discussion questions about the skills Trainability: The participant answers multiple- measured by WSS. A comprehensive list of print- choice questions on a training video covering based scenarios has been designed to foster discussion complex material. and the development of individuals’ competence in Graphs and charts: The participant answers areas measured by the program. multiple-choice questions about information In Indiana, Ivy Tech State College has worked with contained in graphs and charts. the state’s Department of Workforce Development

8 Competencies That Count: Strategies for assessing high-performance skills and a consortium of businesses in a particular com- be used in worksites or training programs as follow-up munity to assess employees’ skill levels and to identify activities to help individuals determine ways to areas of opportunity for collaborative training. The strengthen their behavior in particular areas. WSS feedback reports provided employees with The Lucent Technologies Manufacturing Workforce feedback targeting opportunities for individual’s development. Group reports provided individual Collaborative used Working to assess student learning in a curriculum pilot project that targeted five general organizations with information regarding their workforce as a whole and where training was required skill areas identified as necessary for the manufacturing workforce: personal, interpersonal, systems, proce- to bring groups to higher levels of performance. Finally, reports provided information regarding dures, and information. Fifty-seven incumbent workers participated in the curriculum pilot project, potential areas for collaboration in investing training and development dollars. completing the Working self-assessment as a pre- and post-test. While the population size was too small to generate statistically valid findings on gains/losses in Workplace Success Skills, Learning Resources Inc., 700 any of Working’s nine competencies, the Collaborative Canal Street, Stamford, CT 06902-5921 (203) 637- cited percentile increases in “Responsibility” and 5047; www.learning-resources.com “Teamwork” as indicative of the impact of the training.

Working: Assessing Skills, Habits, and Style, H&H Self-Assessments Publishing Company, Inc., 1231 Kapp Drive, Clearwater, FL 33765 (800) 366-4079; WORKING: ASSESSING SKILLS, HABITS, AND STYLE www.hhpublishing.com

“Working” is a statistically valid self-assessment tool that can be used in a wide range of educational and training settings to assess nine SCANS competencies: Competency Assessment in taking responsibility, working in teams, persisting, a Schools sense of quality, lifelong learning, adapting to change, permanent problem-solving, information processing, NEW STANDARDS PROJECT’S APPLIED LEARNING and systems thinking. STANDARDS

The assessment is administered in 30 minutes and can The New Standards Project, a partnership between the be self-scored. Individuals complete 50 questions, National Center on Education and the Economy and such as “I usually do something I enjoy rather than try the University of Pittsburgh, has designed Applied something different” or “When learning something, I Learning Standards. Part of a package of performance first think carefully about the very best way to tackle standards that includes English Language Arts, Math- it.” For each question, individuals rate the statements ematics, and Science, the Applied Learning Standards as “Almost never like me; Occasionally like me; are one of the most established systems for assessing Moderately like me; Quite a bit like me; or Almost cross-cutting competencies in an academic context. always like me.” Working includes materials that can

Competencies That Count: Strategies for assessing high-performance skills 9 The New Standards Project does not propose that The four other standards describe tools and tech- Applied Learning be a separate area of study but that niques that can be used to complete projects that meet students draw from other subjects to meet the stan- the problem-solving standards. The New Standards dards. The project elected to develop separate stan- partners are clear that these tools and techniques dards, however, to ensure that cross-cutting competen- should not be learned and practiced in a piecemeal cies do not get lost in the demands of more academic way; they must be integrated into projects, and subjects. The five performance standards are: projects should involve subject matter related to the core subject standards. Problem solving; As with its core subject standards, the New Standards Communication tools and techniques; Project defines each standard and delineates perfor- Information tools and techniques; mance descriptions that explicitly describe what students should know and be able to do. The perfor- Learning and self-management tools and tech- mance descriptions are concrete and defined in terms niques; and of actual products, services, or performances that Tools and techniques for working with others. students can use to show evidence of achievement of the standard. The standards are set at the level of the tenth grade, consistent with the partners’ interest in all students The project includes samples of standard-setting across the country completing a common academic student work, collected from more traditional aca- core by the end of the tenth grade and achieving a demic disciplines, as evidence of proficiency in applied Certificate of Initial Mastery. learning standards, and it includes commentary on how the work meets the standards. The work samples The New Standards Project describes the problem- and commentary are detailed, including a description solving standard as the “centerpiece” of the standards. of the task, the circumstances of performance, the It defines projects organized around three kinds of student work, and an evaluation of what the sample of problem solving: student work shows regarding student performance of Design a product, service, or system in which the the standards. Teachers can use the criteria described student identifies needs that could be met by in the Applied Learning Standards to benchmark their new products, services, or systems and creates own students’ work. The project’s cross-referencing solutions for meeting them; system enables teachers and students to identify ways in which projects may be used to show evidence of Improve a system in which the student develops meeting more than one standard in Applied Learning an understanding of the way systems of people, as well as in academic disciplines. machines, and processes work, troubleshoots problems in their operation, and devises strate- The Fort Worth (Texas) Independent School District gies for improving their effectiveness; and collaborated with the New Standards Project by Plan and organize an event or an activity in serving as an implementation site for the pilot New which the student takes responsibility for all aspects of planning and organizing an event or an 2 New Standards Performance Standards, Volume 3: High activity from concept to completion.2 School, p. 106.

10 Competencies That Count: Strategies for assessing high-performance skills Standards in the early 1990s. The district had under- such as SAT scores and grades, as well as more innova- taken a survey of employers and determined that tive measures of student performance, such as judg- many of the skills required in the Fort Worth ments of proficiency. economy were not being taught in the schools, and it Under the leadership of the New Hampshire Depart- had initiated a project to incorporate workplace competencies into academics. ment of Education, the transcript was designed through a collaborative process including representatives from Every summer, a new cohort of 30–40 teachers is secondary schools, post-secondary institutions, business trained in applied learning pedagogies: teachers and the community, in accordance with the state’s develop projects in their content area (i.e., math, curriculum frameworks for career development, English science, or English) that incorporate workplace skills. Language Arts, Mathematics, Science, and Social The school system has done some work developing Studies. Work groups identified a number of important rubrics that map to the Applied Learning standards, features that would characterize New Hampshire’s based on adult models of performance. In addition, transcript. These features included: combining tradi- teachers participating in the summer training are tional reporting measures (grades) and newer compe- encouraged to engage their students in developing tency-based measures of proficiency; aligning the rubrics tailored to each specific project and that take transcript with the state’s academic standards and career- into account both content and the Applied Learning development frameworks; and recognizing a variety of standards. Teachers and students work together to community and workplace learning experiences. determine “how good is good enough?” Students develop their sophistication in identifying, and The state Department of Education issued a Request for Proposals, asking school districts to apply to be “Bench- producing, quality work through participating in the rubric development process. mark Communities,” based on their ability to document evidence of progress in implementing education reform strategies and their interest in piloting the competency- New Standards Project, National Center on Education based transcript. In the spring of 1998, the department and the Economy, 700 Eleventh Street, NW, Suite 750, designated four school districts as Benchmark Communities. Washington, DC 20001 (202) 783-3668; http://ncee,org In the winter of 1998-99, in preparation to use the competency-based transcript in the following school NEW HAMPSHIRE’S COMPETENCY-BASED year, the Benchmark Communities refined the pilot TRANSCRIPT document. They decided to modify the draft transcript to use summative, rather than course-based, judgments. New Hampshire is a leader, among New England In other words, the transcript would reflect a single states and nationally, in designing and piloting a judgment of student proficiency for each designated competency-based transcript. A competency-based competency area at the end of a school year, drawing on transcript is a tool for documenting and assessing evidence from various learning experiences. The pilot skills and knowledge gained through both academic would focus only on documenting student abilities and out-of-school learning opportunities. It can related to five high-performance skill sets or competen- include traditional measures of academic achievement, cies: decision-making and problem-solving skills, self-

Competencies That Count: Strategies for assessing high-performance skills 11 management, communication skills, ability to work high-performance skills from academic courses and with others, and information use (technology, re- out-of-school learning experiences; they were judged search, analysis). Competencies in core academic areas as not meeting, meeting, or surpassing the standards. would be incorporated in later years. Teachers learned to assess portfolio collections of student work against the written performance stan- Work groups, drawn from each of the Benchmark dard for each competency, while students became Communities, took on the task of developing com- acquainted with the performance standards and were mon guidelines for assessing and documenting student coached in collecting evidence of proficiency for their learning in relation to each of the high-performance transcripts. skills. During the spring of 1999, working with Jobs for the Future and a consultant, the groups created a During the 1999-2000 school year, all the high work plan for the pilot. Each of the four schools made schools used the same standards and the same format a commitment to: for the competency-based transcript. Each school developed a plan for internal processes to ensure Pilot the transcript in at least the tenth and consistent judgments among its assessors. Eventually, eleventh grades; these judgments will be determined at each high Include at least 15 percent of the students in each school grade level and at exit from high school. As the grade; pilot evolves, districts will work together to ensure that judgments of proficiency are determined consis- Select a diverse, representative sample of students tently across schools; this will be key to ensuring that and faculty to participate; the transcript provides meaningful evidence of student Offer students a variety of opportunities to learning statewide. collect evidence both within classes and from non-classroom settings; and Ultimately, New Hampshire’s objective is ambitious: to engage public and private post-secondary institu- Prepare all teachers at the pilot schools for tions in using the competency-based transcript in the mapping their courses against the transcript to admissions process, and to involve employers in using show how and where students could collect the transcript as part of their hiring process. As one evidence of mastery of the five high-performance step toward that end, for the 2000-2001 school year, skills. the transcript pilot has expanded, and added ten new In September 1999, the four Benchmark Communi- districts to the original four. In addition, the state has ties began using New Hampshire’s pilot competency- expanded the scope of the pilot transcript to assess based transcript, with a goal of eventual expansion to student performance against proficiency standards in include the academic frameworks and transcript use four core academic areas, outlined in the state’s for post-secondary admissions and employers. Each curriculum frameworks, in addition to assessing participating student became responsible for collecting student high performance skills. New Hampshire’s his or her own work samples as evidence of mastering intention over the next few years is to use the pilot a particular competency, and then presenting this and other methods to expand interest in, and use of, a portfolio to a mentor/advisor or a designated assessor. revised competency-based transcript statewide. Students submitted evidence of proficiency in the

12 Competencies That Count: Strategies for assessing high-performance skills Contact Mariane Gfroerer, Guidance and Counseling student work, with a six-level scale ranging from Consultant, New Hampshire Department of Education, “Distinguished” to “No Evidence.” “Achieves Stan- 78 Regional Drive, Concord, NH 03301 (603) 271- dard” indicates that the student has performed to 6691. grade level; “Mastery” and “Distinguished” indicate that the student has surpassed grade level.

For the past five years, Hoover and another Coalition THE COALITION OF ESSENTIAL SCHOOLS/ school, Homestead High School in Cupertino, have TRANSITIONS PROJECT participated in the Transitions Project, a California- Schools that are members of the Coalition of Essential based initiative to document student learning through Schools network subscribe to ten principles of reform, performance assessments for entrance into the state’s 3 one of which refers to methods of assessment: “Teach- postsecondary institutions. Hoover’s Transitions ing and learning should be documented and assessed transcript documents a student’s performance in each with tools based on student performance of real tasks. subject area in relation to the learner outcomes. For Multiple forms of evidence, ranging from ongoing example, a student might score “Mastery” in the observation of the learner to completion of specific learner outcome “Communication” in the humanities projects, should be used to better understand the course, while scoring “Achieves Standard” in the learner’s strengths and needs, and to plan for further learner outcome “Collaboration” in biology. While assistance…The emphasis is on the students’ demon- individual teachers score portfolio projects, Hoover stration that they can do important things.” used a consensus-scoring process to review student work for the more high-stakes transcripts required by In the process of determining what the “important the Transitions Project. To prepare for this, teachers things” are that students need to know and be able to participated in school-wide staff development on the do, many Essential schools have articulated a set of consensus scoring of student work. learner outcomes that encompass and go beyond traditional academic content mastery. For example, At Homestead High School, three of the school’s Hoover High School in San Diego expects all students “Expected School-Wide Learning Results”—commu- to demonstrate progress on six learner outcomes that nication skills, work skills, and thinking skills—cut are similar to the SCANS skills: habits of inquiry, the across disciplinary boundaries and are developed as experience of technology, collection and organization graduation-level, rather than grade-level, performance of information, communication of ideas, effective expectations. Assessments of these outcomes are cooperation with others, and lifelong learning. designed to reflect students’ progress in moving from

Hoover students learn these skills as they move 3 through ninth/tenth-grade interdisciplinary courses Several other high schools participated in the project, a collaboration among the California Center for School and eleventh/twelfth-grade “institutes” (or career Restructuring, Pacific Education Group, and the Bay Area pathways). Career exploration and community Coalition of Essential Schools. Only Hoover and Home- stead and one additional San Diego high school, Rancho learning activities are a feature of all four years. Buena Vista, developed rubrics and participated in the A school-wide rubric that contains performance process of submitting alternative transcripts to the Califor- descriptors for each learning outcome is used to assess nia public university systems throughout the life of the project.

Competencies That Count: Strategies for assessing high-performance skills 13 “Novice” to “Exceeds Standards” in relation to a given decision making, research, and the use of knowledge standard. For example, while a ninth-grade student and applications from many disciplines. Project might be expected to score at the “Novice” level, s/he developers identified problems, wrote the stories, would be expected to progress to “Proficient” by included data (and gaps in knowledge), developed graduation. The fourth Learning Result relates to characters and dialogue, and developed the CD- standards that are specific to course content, and these ROMs to include simulated offices. are assessed separately according to grade-level expectations using a four-level scale, from “Novice” to Students work in teams to solve the problems, each of which addresses one or two primary SCANS skills as “Exceeds Standards.” well as specific academic content. The CD-ROM Homestead High School created, but later suspended, provides enough data to get started and carefully integrated studies programs in grades 9/10 and 11/12, placed tutorials on such topics as linear programming, organizing them around essential questions or prob- statistics, and simulated flow-charting. Periodically, lems. Because the integrated studies programs were students are directed to stop the CD and complete not school-wide, they involved only a subset of activities. Students may arrive at solutions contained teachers and students in submitting alternative within the CD-ROM, or they may develop alternative transcripts under the Transitions Project. However, solutions not envisioned by the project developers. other teachers, across the school, use the Transitions Project rubric. Because not all teachers use the perfor- To assess student learning, SCANS 2000 surveyed the field to identify tools for assessing some of the “hard mance assessments, Homestead’s transcript includes both performance assessments, where available, and to measure” skills, such as teamwork, that the CD- ROMs emphasize. The effort has selected two mea- course grades. sures, one for diagnostic purposes and one for standardized assessment, and opted to pilot these in three For more information, see “New College Admissions SCANS skills: teamwork, interpretation and use of Procedures: Implications for Career-Related Learning in information, and problem-solving. High School,” National Center for Research in Vocational Education, August 1999; or go to the Coalition of For diagnostic assessment, SCANS 2000 turned to Essential Schools Web Site: www.essentialschools.org. AES International, a company founded in 1993 for the purpose of providing tools in support of the recently released SCANS skills. SCANS 2000 worked SCANS 2000 CENTER/AES INTERNATIONAL “SKILL collaboratively with teachers and AES International to COMMAND” develop assessment rubrics from AES’ “Skill Com- mand,” an assessment instrument in which various SCANS 2000 at The Johns Hopkins University and people (e.g, teachers, peers, employers, self) rate collaborators have developed CD-ROM case studies participants’ behaviors. The assessment rubric breaks designed to help community college students master each of the major skills into multiple behaviors on SCANS skills within their academic coursework. The which teachers rate students on a scale from “No case studies simulate real workplace problems: they are Understanding” to “Expert.” For example, the rubric “messy” situations requiring sustained effort, complex lists seven behaviors under problem-solving, ranging

14 Competencies That Count: Strategies for assessing high-performance skills from “Clearly Defines the Problem” and “Considers “authorizes” the information to be posted to the Alternative Solutions” to “Monitors Progress of transcript, where employers can view it. The result is a Proposed Solution.” The instrument also specifies transcript, similar to a résumé in that it shows past when to measure students on each of the skills. For job-relevant experience, yet different from a résumé in example, students are assessed on problem-solving at that the employer is assured that the information is the same time they are assessed on solving the validated through an objective source. These tran- advertising budget, using the CD-ROM, in an scripts are expected to facilitate entry into the algebra class. The “teamwork” skill assessment is workforce and provide a platform for documenting anchored to the task through which team members developmental change throughout a participant’s must select a car to market, and it assesses students’ career. cooperation with and contribution to their team. Each task is rated with a SCANS level of difficulty For more information about Skill Command, go to because, for example, problem solving in one venue www.skillcommand.com. To learn more about SCANS may be more complex than in others. 2000, go to www.scans.jhu.edu.

For a more objective pre- and post-assessment, the team selected the Work Success Skills video simulations, profiled above. It selected the WSS system for Competency Assessment in the three reasons: the WSS videos measure the skills Workplace emphasized in the CD-ROMs; it is objective and industry-validated, predicting job success four times The Massachusetts Work-Based Learning Plan was more accurately than an interview; and it generates developed for the use of employers offering work data on student performance that includes informa- experiences to high school students. It is a tool for tion on strategies a teacher or the learner can use to assessing student learning of cross-cutting competen- address specific skills gaps. cies in the workplace. The other two assessment methods described below—SHL Assessment Centers The SCANS 2000 team is piloting the assessment and Behavioral Interviewing—are used in the work- process with community college partners and will place to identify and develop entry-level and execu- work with the faculty to expand the number of tasks tive-level employees; these have not yet been imple- and skills they will assess. Meanwhile, the SCANS mented in schools. Both are included here because 2000 team, in partnership with AlignMark, has they focus on training assessors to use observational developed a Career Transcript System (CTS) that is skills to assess performance. School systems might being piloted throughout the country in high adapt this methodology to assist teachers in using schools, universities, and social-service agencies. The observational skills to identify ways in which students CTS tracks and documents the attainment of are, or are not, gaining high-performance skills in SCANS skills. The participant takes part in an accomplishing school-related tasks. assessment such as Work Success Skills or Skill Command; the assessment results are posted to the CTS; and if the participant likes the data then s/he

Competencies That Count: Strategies for assessing high-performance skills 15 MASSACHUSETTS WORK-BASED LEARNING PLAN Team Competencies: acting professionally; interacting with others; understanding all In 1996, the Boston Private Industry Council (PIC) aspects of the industry; and created a Work-Based Learning Plan to enhance the quality of work-based learning in its programs that Personal and Professional Development Compe- link academic and career-related classes with work- tency: taking responsibility for career and life based learning. The PIC revised occupation-specific choices. training plans to focus instead on broad competencies Each of the nine competencies is further divided into that Boston employers, schools, and community more specific skills. The Work-Based Learning members agreed were essential to success in higher Statewide Steering Committee developed a rubric education and the world of work. The learning plan is that specifies ratings for each skill, ranging from a master form that students, teachers, worksite super- “Needs Development” to “Advanced.” For example, visors, and PIC staff complete for each placement. It under the speaking skill (communication and literacy focuses students, employers, and school-based person- competency), a student rated “Needs Development” nel on a mutually agreed-upon set of skills and is “learning to speak clearly, audibly, and courte- abilities that students should have the opportunity to ously;” a student rated “Proficient” “expresses com- develop. It also provides an opportunity for teachers plex ideas in a well thought-out, organized, and and workplace supervisors to engage together in concise manner.” Under the research and analysis skill developing the skills and competencies of individual (organizing and analyzing information competency), students, allowing for a greater connection between a student who “Needs Development” is “developing classroom and workplace learning. familiarity with pertinent information;” an “Ad- The Massachusetts Office of School-to-Career, vanced” student “develops theories of action and tests through a statewide Work-Based Learning Statewide them in practice.” Steering Committee, has adopted the Work-Based The learning plan asks supervisors to undertake a Learning Plan for use across the state, and has created three-step process: an electronic version of the Plan along with a guide for supervisors on its use. The Massachusetts Office of 1 Use the plan to identify the job description and School-to-Career has also released a toolkit to help tasks to be completed, then pick the three to five trainers across the state train employers, educators, competencies most critical to the job. students, and parents on how to ensure an effective 2 After observing the student’s on-the-job perfor- work-based learning experience. The toolkit contains a mance for a week or two, conduct an evaluation CD-ROM with reproducible materials from the with the student and set goals in competency toolkit, including the Work-Based Learning Plan. areas.

The learning plan covers nine competencies of three 3 At the end of the work experience, or at appro- overall types: priate intervals during longer placements, conduct a follow-up assessment meeting with Individual Competencies: communication and the student. literacy; organizing and analyzing information; problem solving; using technology; and completing entire activities;

16 Competencies That Count: Strategies for assessing high-performance skills Instructions for the learning plan encourage supervi- tions about what an applicant did are better predictors sors to complete the ratings with the student so that of behavior than questions about what an applicant students develop an awareness of their own skills levels would do. Behavioral interviewing focuses less on and an understanding of the learning potential of the content or technical knowledge and more on how an workplace. applicant accomplished activities in previous employment. The PIC has adapted the learning plan for alternative education initiatives, such as Diploma Plus, a program Often, businesses use consultants to help them for at-risk and out-of-school student participants that identify critical competencies for particular jobs and requires at least 150 hours of paid or unpaid work or to train their staff on behavioral interviewing tech- community service. It has also introduced the learning niques. In this way, it is also a more efficient means of plan and its competencies to job shadowing experi- interviewing than most: interviewers can assess for ences: employers can use the competencies to show particular skills, rather than use a broader assessment students the skills needed for the workplace, while instrument to assess all cross-cutting competencies. teachers can use them in the classroom as a “checklist” Typically, a business will identify the competencies that gives students an analytic framework to reflect upon their brief experiences and compare them to past demonstrated in effective performance of a job, then develop job-related questions to gather examples of or future shadows. related past activities. For example, to probe for an applicant’s attention to detail, an interviewer might Massachusetts Work-Based Learning Plan, Massachusetts ask the applicant to give an example of when s/he had Department of Education, 350 Main Street, Malden, to deliver a finished product under a tight deadline. MA 02148 (781) 388-3382; www.doe.mass.edu/stw/ The trained interviewer listens for clarification, paying mwblp.html. This web page has many tools and materials attention to what descriptive words the applicant uses on the implementation of the work-based learning plan. to describe the event. For example, the interviewer might say:

BEHAVIORAL INTERVIEWING What were you thinking?

In behavioral interviewing, a strategy widely used by What were you feeling? businesses, interviewers elicit actual examples of past What was your contribution to that? behavior in order to predict future performance. Most often, the skills for which behavioral interviewers Walk me through your thought process. probe are “soft” skills, such as attention to detail, Take me through that step by step. leadership, initiative, and decision making. As one private-sector user of this methodology put it, “Tech- How did you know what to do? nology skills are easy to measure. We use behavioral Afterward, the interviewer codes the interview, often interviewing to help us identify which applicants have using a rubric developed specifically for the identified the leadership and organizational capabilities to competencies. succeed on the job.” Its proponents argue that ques-

Competencies That Count: Strategies for assessing high-performance skills 17 The Bank of America has adapted the behavioral Tools include: interviewing process with its welfare-to-work program. Group exercises with assigned roles, in which In recognition of former welfare recipients’ previous every participant is given a different set of experience, the bank’s human resources staff are background materials; trained to elicit examples of past performance in the home as well as in the workforce, and partnering Group exercises with unassigned roles, in which community-based organizations train applicants who all participants receive the same background have been on welfare to develop examples from both materials; work and home experience to identify competencies. “In-tray” exercises, in which participants must analyze and prioritize written documents and The behavioral interviewing process contains clues as propose actions; to how teachers might enhance their assessment of individual or group projects. For example, a teacher Fact-finding exercises, through which an indi- could probe for specifics about what students did and vidual starts with partial knowledge and must thought in producing a report or project, or might ask decide what extra information is required to students to clarify their individual input into a group complete the tasks, often with a “resource per- effort. Student projects are often assessed solely son” available to provide additional information as requested; and through the presentation of a finished product: behavioral interviewing is a way to make visible the Analysis-presentation exercises, in which partici- often invisible learning in problem-solving or team- pants must absorb and analyze information, then work gained through projects. present their conclusions and recommendations for action. SHL ASSESSMENT CENTERS For the group exercises, participants receive materials SHL, an international human resource consulting (e.g., site plans, sales figures, staffing requirements) firm founded by industrial psychologists in 1977, and have 10 minutes to prepare for the exercise helps organizations assess, select, and develop employ- individually. The group then has 50 minutes to ees at all levels. SHL has developed an array of “assess- determine a solution to the given problem, such as ment centers.” These are simulations of employee determining the allocation of space in a new office or tasks and accompanying tools to assess how individu- manufacturing space. Trained assessors observe the als address work-related challenges, such as function- participants’ actions and rate their performance ing in group and team settings, organizing and according to a scoring rubric. For example, to assess prioritizing critical tasks, and conducting research, “influence,” assessors must determine if tasks are analysis, and idea generation to solve problems. delegated appropriately; to assess “team membership,” Assessment centers may be job-specific, or they may assessors determine if all suggestions of the group are assess for skills across a variety of positions. considered or dismissed out of hand.

18 Competencies That Count: Strategies for assessing high-performance skills The simulations have been used in a wide variety of Competency Assessment in Adult settings for both hiring and staff development pur- Education Settings poses. Because it can be costly to train assessors, the simulations tend to be used for senior management EQUIPPED FOR THE FUTURE positions, although companies have used them to identify promising college graduates. For example, The National Institute for Literacy, an independent financial institutions have used SHL simulations to federal organization, has developed a set of learning recruit financial analysts; a major pharmaceutical standards that are based on the knowledge and skills corporation has used them to hire sales staff, and a that adults need in order to carry out their roles and major chemical corporation has used them as a responsibilities as workers, parents and family members, developmental tool for its managers around the world. and citizens and community members. Through a A major national insurance company conducted a collaborative and grassroots process involving adult “high fidelity simulation” in which applicants viewed education initiatives across the country, the Institute has a video of the performance of the desired job, partici- identified skills and knowledge that are critical to family, pated in a tutorial on the fundamentals of the civic, and work life. Equipped for the Future (EFF) company’s customer service software, and then were Standards for Literacy and Adult Lifelong Learning were assessed as they responded to simulated calls from developed through a careful research process with adult customers asking for assistance. learners, resulting in a core set of 16 content standards that include workplace competencies but encompass the Educators interested in experimenting with the complexities of negotiating civic and family life as well. assessment center process for their classroom might adapt a simulation that is not occupation-specific. For The Equipped for the Future skills are as follows: example, they might assess students’ skills in gathering Communication Skills: Read with understanding, and analyzing information, prioritizing tasks, or convey ideas in writing, speak so others can under- making presentations. stand, listen actively, observe critically;

SHL North American Headquarters, Flatiron Park West, Decision-Making Skills: Use math to solve problems and communicate, solve problems and make 2555 55th Street, Suite 201D, Boulder, CO 80301 decisions, plan; (303) 442-5607; www.shlusa.com Interpersonal Skills: Cooperate with others, advocate and influence, resolve conflict and negotiate, guide others; and

Lifelong Learning Skills: Take responsibility for learning, reflect and evaluate, learn through research, and use information and communications technology.

Competencies That Count: Strategies for assessing high-performance skills 19 In developing the standards, collaborators started with teachers decided to use this activity to teach the skill the question of what adult “competence” looks like, “plan,” one of the decision-making skills. paying attention to the range of adult learners from those with basic skills to those with formal education First, Even Start teachers reflected on the degree to which the students already had the skill of planning; and experience. They have identified four key dimensions of performance to generate detailed descriptions of they determined what to look for in order to know if the students were attaining this skill, focusing on the learner performance. Performance dimensions will eventually be used for placing descriptions on a four dimensions of standards, described above—for example, can students plan independently? Can they developmental performance continuum for each EFF standard. These dimensions are: plan under a range of conditions? Then, the teachers explicitly taught the steps to planning, using the Depth and breadth of knowledge base; standard’s components of performance as a guide. With teacher guidance, students developed the plan Degree of independence; for accomplishing the task of creating the sweatshirts. Range of conditions under which one can After students carried out the activity of creating the perform; and sweatshirts, they held discussions to evaluate the activity and their planning process. Students then Degree of fluency and flexibility with which one completed a written self-evaluation, which they placed can perform. in their portfolio. The National Institute for Literacy is using a The assessment of student learning was based on grassroots approach to developing the EFF assessment teachers’ ongoing observation of student performance framework. The institute is working with adult in this activity and in other arenas of their lives. For education programs in five states (Maine, Ohio, example, the teachers provided follow-up activities to Oregon, Tennessee, and Washington) to identify a determine if their students had learned how to plan, research-based performance continuum for each EFF which also gave students the opportunity to transfer standard that will support the identification of level skills to another context. In one such activity, students descriptors for all 16 standards. Participating teachers had to complete an individual written plan before are collecting information about student performance working on creating a “literature kit” for their chil- and progress against the content standards, using a dren, rather than developing a plan as a group with data collection template that focuses observation and teacher guidance. Teachers used this activity as a post- documentation of learner performance on the four test to identify whether students could integrate all the dimensions of skill performance. skill’s components and in what areas students needed Currently, the most common way that adult education further skill development. Teachers also pointed out to practitioners are assessing the EFF standards is students when they were successfully integrating the through the use of portfolios. For example, at the skill into their daily lives, such as when they had to Even Start Program in Canton, Ohio, adult learners plan to secure day care arrangements. This helps participated in a joint project with a hospital, decorat- students to reflect on their own skill attainment and ing sweatshirts for donation to a local charity. The to convey it to others.

20 Competencies That Count: Strategies for assessing high-performance skills Along with other student work, the written plans for Conclusion each activity were included in their portfolios. Stu- dents’ accomplishments, documented in the portfo- Interest in assessing high-performance skills is coming lios, form the basis for “Career Passports” that stu- from many quarters: employers, state colleges, and dents use for employment searches. The Career universities that reside at the junction between educa- Passport summarizes a student’s marketable skills and tion and the economy; high school educators seeking includes a résumé, awards or certificates, a list of the to expand beyond traditional curricula and outside the person’s competencies, and letters of recommendation. school walls; and parents concerned that their children get critical thinking skills necessary for success in For more information about Equipped for the Future, postsecondary education and in high-skill careers. contact the National Institute for Literacy, 1775 I Street, This guide can serve as a starting point for those who NW, Suite 730, Washington, DC 20006, (202) 233- seek to equip young people with a broad range of 2025; www.nifl.gov. Equipped for the Future publications cross-cutting, high-performance skills that will help are available from ED Pubs: 1 (877) 433-7827. For them succeed in postsecondary education and in the more information about the Even Start program, contact changing economy. Jane Meyer, Adult Basic and Literacy Education Coordi- nator, Canton City Schools, (330) 588-2148, ext. 130.

Competencies That Count: Strategies for assessing high-performance skills 21 Northeast and Islands Regional Educational Laboratory a program of The Education Alliance at Brown University

The LAB, a program of The Education Alliance at Brown University, is one of ten educational laboratories funded by the U.S. Department of Education’s Office of Educational Research and Improvement. Our goals are to improve teaching and learning, advance school improvement, build capacity for reform, and develop strategic alliances with key members of the region’s education and policy making community.

The LAB develops educational products and services for school administrators, policymakers, teachers, and parents in New England, New York, Puerto Rico, and the Virgin Islands. Central to our efforts is a commitment to equity and excellence.

Information about LAB programs and services is available by contacting:

LAB at Brown University The Education Alliance 222 Richmond Street, Suite 300 Providence, RI 02903-4226

Phone: 800/521-9550 E-mail: [email protected] Web: www.lab.brown.edu Fax: 401/421-7650

Jobs for the Future (JFF) is a LAB partner organization. JFF is a national non- profit organization that works to strengthen the foundation for economic opportunity and civic health in America by advancing the understanding of the skills and knowledge required for success in the new economy. JFF works locally and nationally to develop innovative workforce development solutions that help people make effective lifelong transitions between work and learning.

This publication is based on work supported by the Office of Educational Research and Improvement (OERI), U.S. Department of Education, under Contract Number RJ96006401. Any opinions, findings, and conclusions or recommendations expressed in this material are those of the authors and do not necessarily reflect the views of OERI, the U.S. Department of Education, or any other agency of the U.S. Government.

Adeline Becker Executive Director, The Education Alliance

Phil Zarlengo Executive Director, The LAB at Brown University

Vincent Ferrandino Chair, LAB Board of Governors

Marjorie Medd Vice Chair, LAB Board of Governors

Board Members

J. Duke Albanese Richard Mills

Barbara Bailey Thong Phamduy

Pamela Berry Daria Plummer

Paul Crowley Olga Lucia Sallaway

David Driscoll Theodore Sergi

Katharine Eneguess David Sherman

Victor Fajardo Ruby Simmonds

Charlotte K. Frank Jeanette Smith

Harold Levy Jill Tarule

Edward McElroy Elizabeth Twomey

Peter McWalters David Wolk LAB Northeast and Islands Regional Educational Laboratory

a program of

222 Richmond Street, Suite 300 Providence, RI 02903-4226

Phone: 401/274-9548 Fax: 401/421-7650 E-mail: [email protected] Web: www.lab.brown.edu

LAB Working Paper No. 2