Evaluation Forum

Youth and the Postindustrial Future

U.S. Department of Labor Employment and Training Administration

Issue 12, Summer 1997 Evaluation Forum

Youth and the Postindustrial Future

Issue 12, Summer 1997

U.S. Department of Labor Secretary, Alexis M. Herman

Employment and Training Administration Acting Assistant Secretary, Raymond J. Uhalde

Office of Policy and Research Gerard F. Fiala, Administrator

A review journal for employment and training professionals focusing on policy analysis and evaluation research in occupational training, education, work and welfare, and economic development.

Evaluation Forum n Issue 12 n Summer 1997 i This issue of Evaluation Forum is the last in the current series of journal issues sponsored by the U.S. Department of Labor. The Office of Policy and Research is in the process of considering options for the future.

Any ideas, comments, or suggestions you may like to share with the editor or the sponsor should be directed to:

Eileen Pederson Office of Policy and Research Employment and Training Administration U.S. Department of Labor

fax: 202. 219.5455

ii Evaluation Forum n Issue 12 n Summer 1997 Table of Contents

1 American Social Policy Features An analysis of economic and sociocultural issues and trends, and their impact on American poverty policy, as a context for studying youth poverty. 5 A Generational Perspective on Young Adults Commentaries on critical differences between two generations of youth: youth of the World War II era, and youth of the 1980s and 1990s. 9 Rising Income Inequality and Poverty An analysis of the effect of global and domestic economic and technological change, family structure and poverty policies on economic equality and job opportunities for youth. 14 A Victim of Policy Neglect? Wealth Inequality A study of a long neglected issue affecting the life chances of American youth — rising wealth inequality. 17 Balancing Growth with Economic Equality A provocative analysis of the relationship between slow economic growth and increasing inequality. 18 Economic Inequality: A Comparative View A report on some of the striking results of the Luxembourg Income Study, based on data collected across twenty-five countries.

21 The commentaries of a panel of experts on youth development, education, Commentary training, and employment policies and programs: David Brown, National Governors Association Gary Burtless, The Brookings Institution Susan Curnan and Andrew Hahn, Brandeis University Robert Ivry, Manpower Demonstration Research Corporation Chris King, University of Texas Gary Walker, Public/Private Ventures, Inc. Alan Zuckerman, National Youth Employment Coalition

38 An Interview with William Julius Wilson Interview An interview with William Julius Wilson of Harvard University, based on his new edited book, Poverty, Inequality, and the Future of Social Policy. The interview focuses on Wilson’s research-based perspective on the implications of increasing employment challenges, growing income and wealth inequality, and rising poverty on the life chances of youth.

Evaluation Forum n Issue 12 n Summer 1997 iii Evaluation Issues International Perspectives on Youth Issues 43 An Overview of Research on International Youth Issues and Activities An analysis of the current context of youth training and employment as it relates to economic and societal development and politics across industrialized and developing nations. 45 Youth Training A study of the training and labor market outcomes of youth in the U.S., Great Britain and Australia, based on three national longitudinal surveys. The Performance Measurement Movement 49 Performance Measurement and Continuous Quality Improvement A policy analysis focusing on the relationship between continuous quality improvement, a component of strategic planning, and evaluation research. 50 Challenges in Developing Performance Measures and Standards: The Case of Educational Skills Several exemplars illustrating the problems and possibilities in developing performance measures and standards. 52 Performance Measurement Systems and the National Performance Review A review of the major features of the Vice President’s National Performance Review, and its implications for strategic planning and evaluation. 56 Testing the Government Performance and Results Act Lessons learned in testing the national Government Performance and Results Act of 1993. 59 The Experiences of Other Countries with Performance Measurement The General Accounting Office’s study of performance measurement in Australia, Canada, New Zealand and the United Kingdom. 61 Performance Auditing and Evaluation Research An analysis of the auditing world’s new movement into performance auditing, tied to budgeting. 62 More Insights about Performance Auditing and Evaluation Research A review of an interview with the Financial Controller of the European Commission on the relationship between performance auditing and evaluation research. 64 Comments on the Performance Measurement Movement in Australia A perspective on Australia’s definition of performance management. 66 Comments on the Performance Measurement Movement in the Netherlands A viewpoint on the Netherlands’ version of performance management. 67 Improving Performance Systems in Supporting Program Goals A review of selected studies of performance management systems and their trade-offs, with recommendations for improvement. 70 The Potential Effects of Performance Standards Systems An extensive summary and analysis of recent studies on the effect of performance standards on the achievement of social program goals.

iv Evaluation Forum n Issue 12 n Summer 1997 Alternative Evaluation Approaches 78 Testing the Use of Program Applicants in Studying the Net Effects of Social Programs A guest review by economist Ernst Stromsdorfer of a new book dealing with the potential effects of performance standards on the implementation of the intended goals of social programs. 80 Participatory Mixed-Method Evaluation A discussion of the advantages of considering a range of research designs and methods for studying youth issues and programs, based on a methodological approach used by the Tavistock Institute in London. 83 The European Commission’s Effort to More Effectively Structure Evaluation Activities A review of an evaluation guide developed by the European Commission for use in planning and monitoring evaluation activities, and judging the efficiency and effectiveness of European Union social and economic development programs.

87 U.S. Youth Initiatives and Program Evaluations: Selected Examples Resources A review of selected evaluations of youth programming. 90 Useful References on Youth-Related Issues and the Evaluation of Youth Programs A listing of analyses and studies by issue area, including selected international references.

Evaluation Forum n Issue 12 n Summer 1997 v vi Evaluation Forum n Issue 12 n Summer 1997 Features Features

nn Editor’s Introduction expectations defined what it meant to be American. . . . It empowered people American Social Robert J. Samuelson, a columnist to think they could shape their own Policy for the Washington Post, talks about destiny.” However, in the past twenty American unhappiness in the midst of years the rate of economic progress American success in his new book The has slowed beyond any era since the Important pieces of the fabric of Good Life and Its Discontents:The Civil War. The output of goods and ser- complex designs against which youth American Dream in the Age of Entitle- vices produced per hour of work has issues must be analyzed are provided ment. He claims we transformed the also slowed. Furthermore, although in Ronald B. Dear’s article on Social American Dream into the American rapid growth tends to benefit everyone, Welfare Policy: Trends and Issues in the Fantasy. A reasonable demand for slow growth apparently does not. Given new three-volume Encyclopedia of So- progress, he says, has become an ex- that the rate of economic growth and cial Work published in 1995. Dear is a cessive demand for perfection. Conse- productivity are the main determinants specialist in social welfare policy on the quently, “our attitudes are shaped more of our living standards, these changes faculty of the University of Washing- by unattained ambitions than actual are shaping a new public malaise. ton’s Graduate School of Social Work, achievements.” He insists that by most Madrich concludes that this is a signifi- but his exposure to policy issues is not standards American society is better off cant reversal of our national experi- limited to academia. As the liaison be- than a half century ago. Does this au- ence and buoyancy. tween the University and the Washing- thor understand what’s really happen- This blend of optimism and pessi- ton State Legislature, and later as the ing to American society? There is no mism in the mid-1990s is affecting the chair of the University’s Faculty Senate, question that many facets of American economic, socio-cultural, political and Dear has experienced up close the life have grown better over the past fifty value environment of this generation of machinations involved in developing years. But the results of competent re- adolescents and young adults, influ- and implementing policy in a highly po- search lead us to have some qualms encing their life chances, framing their liticized environment. Never wavering about our overall “progress” as an attitudes and behavior. from a dual interest in root causes as egalitarian social democracy. In this issue of Evaluation Forum we well as social policy strategies, his anal- Writing in the New York Times op-ed focus on American youth, and on the yses are driven by the need to under- column in January, 1996, author of The historical and contemporary forces that stand the underlying forces shaping so- End of Affluence, Jeffrey Madrick, has are creating a different society for them cial welfare policy in the United States. a slightly different perspective on soci- than the one earlier generations have This commitment helps frame a broad etal history. He acknowledges that al- experienced. This is a changed society, approach to youth issues, particularly though the income gap has widened in which young people will become so- given the increase in youth poverty and and real wages fallen, most people are cialized, will mature, will seek to its correlates. better off than their grandparents or achieve developmental tasks, and will Beginning with a definition and in- great-grandparents. But he maintains become the decisionmakers of the so- terpretation of social welfare policy, that this comparison overlooks the sig- ciety’s future. Dear gives us an historical perspective nificant historical shift in the economy on American social policies that have and public attitudes that provides a Since we can address only a few of attempted to support equity and justice reasonable rationale for the anxiety re- the many issues relevant to the matu- for all citizens. He identifies the major flected in American public opinion ration of youth into adult roles, we fo- influences and trends that have helped polls about the state of the union. cus in the Features Section on the darker side of American society --that form such policies, and suggests proba- Most workers did not live as well be- is, on increasing economic insecurity ble future directions. His analysis is fore World War II as they do now, he and inequality -- since many of our concerned with the entire public social says. But over the course of their lives youth education, training and employ- welfare system, including income sup- they made extraordinary gains in their ment programs respond to such basic port, health care, education, employ- standard of living. In this context, Mad- societal problems. ment and training, and social services. rich points out that “exceeding one’s He contends that youth problems de- manding policy action must be concep- tualized comprehensively, within the broader context of the larger social wel- fare system and the society.

What Is Social Policy? The author explains social policy as a set of principles, plans, procedures and courses of action that set parameters around government action — what gov- ernment can and cannot do. These are

Evaluation Forum n Issue 12 n Summer 1997 1 Features

expressed in legislation, administrative burn’s work had supported the conclu- the resource allocation process would rules and regulations, judicial decrees sion that social trends develop a force- solve unemployment and poverty. and organizational agendas. Social poli- ful momentum of their own and there- Reagan’s New Federalism converted cy involves basic assumptions and fore resist efforts to reverse them. The such theories into pragmatic action by guidelines that determine the form gov- Committee’s report reiterated that posi- consolidating programs, reducing the ernment activity takes in various sub- tion, emphasizing the trend toward funding, decentralizing, and, whenever stantive areas. In this sense, Dear greater government control in the inter- possible privatizing. “Devolution” was views public social policy as a tool or est of securing the common good. The expected to encourage more efficiency mechanism used by government for allo- report also encouraged government to and effectiveness and to interfere less cating limited resources, based on sev- collect, analyze and use quantitative with economic productivity and devel- eral premises: time series data as a support for policy opment. Block grants became one of development and planning. This the strategic mechanisms for acting on n Public economic resources, as well launched a historical effort to study the these premises. as most natural resources, are almost relationship between social trends and always in limited supply. Using federal budget data, the Cen- emerging social problems. ter on Budget and Policy Priorities n Individuals and societies have almost Thirty years later there was another showed that between fiscal years 1981 unlimited needs. surge of interest in social trend analy- and 1987 Community Development n Resource allocation processes are sis. Dear refers to a 1964 report by the Block Grants were reduced by 39%, never fully equitable or just. Societ- U.S. Department of Health, Education Community Services Block Grants by ies have built-in unfairness. and Welfare (HEW) that demonstrated 36%, and Social Services Block Grants how demographic trends and trends in by 28% (after inflation). The nation’s n No consensus exists about how eq- education, employment, income and principal source of federal funding for uity and justice in resource distribu- health care could converge to produce child care and child protective services tion are to be achieved. new social problems. A subsequent was Title XX of the Social Security Act, The author concludes that disagree- HEW report in 1969 attempted to show passed in 1975. Title XX required state ment and conflict over the values to be the utility of an ongoing analysis of so- matching funds, targeted funds to low honored in allocation processes are in- cial indicators for identifying the evolu- income citizens, and mandated public evitable. Since compromises involved in tion of major social trends. Daniel Bell, participation in the development of making allocation decisions tend to co-chair of the government’s Panel on state plans and social services. When foreclose consideration of other options, Social Indicators, viewed the report as folded into a block grant in 1981, these Dear feels it is essential that the proba- the first bona fide effort to measure the requirements were removed, social ser- ble impact of particular value tradeoffs performance of the society in meeting vice funds were cut, and most report- be evaluated. In saying so, he is inter- human needs. ing ended. Also there were sharp re- ested in what forces are most likely to ductions in transfer payments and Unfortunately the Vietnam War and contribute to progressive social change. in-kind benefits, and more restrictive el- reduced social spending diverted atten- igibility requirements. For example, in Among the influences affecting so- tion from this approach to social policy just two years Unemployment Insur- cial democratic change, Dear highlights development. Interest in an annual so- ance was reduced 17.4%, child nutri- economic scarcity in the public sector, cial welfare “report card” waned. And tion programs by 28%, Food Stamp ex- distrust of government, lack of agree- as the movement toward a more re- penditures by 13.8%, and AFDC by ment on a politically-sustainable social sponsive and generous government so- 14.3%. policy agenda, the absence of a political cial welfare policy slowed, this in itself vision and the will to support innova- became a major new social trend. Dear The author sees the combination of tive approaches, and the territoriality makes the case that the twenty years Reagan’s tax cuts and increased de- and pressures exerted by special interest between 1981 and 2001 may someday fense expenditures during a recession- groups. While these forces have compli- be perceived as a turning point in social ary period as the beginning of our cur- cated the development of remedies to welfare policy, an era of increasing but rent obsession with federal revenue significant social problems in the U.S., less well met needs, of lost social wel- shortfalls. The annual federal deficit Dear reminds us that since World War fare opportunities, and of dampened and debt soared, growing even larger II we have been moving steadily in the social innovation. under Bush. In the context of deficits direction of European “welfare states,” and debts of unprecedented magni- The belief that government is the though with considerable ambivalence tude, significant new social welfare ini- main institution responsible for the and hesitation. tiatives lost support. Moreover, nearly well-being of less fortunate citizens, all existing federal social programs be- represented by the New Deal and Great came suspect and vulnerable to reduc- A Selected History of Attempts to Society programs, was rejected in the tion, consolidation or elimination. Nev- Understand and Document Major Reagan-Bush period. Dear points out ertheless, Dear reminds us that the that conservative writers such as Mur- Social Trends large social insurance programs, such ray (Losing Ground) and Gilder (Wealth Dear highlights Hoover’s appoint- as Social Security and Medicare, sur- and Poverty) expounded a “new” theo- ment of William F. Ogburn, a noted so- vived, leaving the core of the public ry, that government social programs ac- ciologist and Director of Research for American social welfare enterprise in- tually increased rather than ameliorated the President’s Committee on Social tact. However, the 1994 elections repre- social problems. Supply-side economic Trends. Recent Social Trends, the mam- sented the first major effort to revise theory, Dear says, only confirmed the moth report of the committee, was per- this core, with the purpose of propos- notion that shrinking the size of govern- haps a defining moment in American ing a very different approach to re- ment and precluding its intrusion into social welfare policy development. Og- source allocation.

2 Evaluation Forum n Issue 12 n Summer 1997 Features

Important Social and Economic vancement. Although a larger propor- new home and business construction, Trends Affecting American Social tion of Americans attend college than and new organizational structures to in most other countries, Dear cites evi- support these changes. The aging of Welfare Policy dence that lower-income and some mi- the “baby boom” generation and other Dear emphasizes an array of changes nority young people are considerably un- demographic changes are important in American society that have influ- derrepresented in American higher also. In 1940, only 6.8% of the popula- enced the way we view social problems education. This underrepresentation tion was over 65; by 1972 it was over and try to solve them. may be due to a number of factors, 12%. By 2030, over 20% of the U.S. such as inadequate preparation, high population is projected to be over 65. Economic Change cost, inaccessibility, discrimination, or The youth population, whose adolescent Dear agrees with most social policy the pessimistic view on the part of portion declined in the 1980s, is expect- experts that although the U.S. gener- these youth that no strategy is likely to ed to grow slowly through the 1990s. lead to mainstream economic success. ates approximately $6 trillion per year Those born between 1946 and 1964 in wealth, and has the world’s highest constituted up to that time the largest level of productivity, a large segment of Demographic Change baby cohort in U.S. history, represent- the population does not share in this af- Between 1930 and 1975 the popula- ing 30% of the U.S. population in 1992. fluence. Labor productivity has slowed tion of the world doubled. Between Dear views the size of this cohort as dramatically and real earnings have de- 1930 and 2000 the world’s population the predominant demographic event af- clined. Both low-technology and high- will have tripled -- it is projected to fecting social policy in this era. He technology products once manufac- reach 6 billion by 2000 and 8.5 billion places the generation born between tured in the U.S., supplying jobs for by 2025. At the current growth rate, the 1925 and 1942 between the politically U.S. workers, are now frequently man- population will increase as much in powerful older generation and the nu- ufactured overseas and imported back twelve years as it did in the first three merically powerful baby boomers. Born into the U.S. Meanwhile, improved effi- million years since the beginning of hu- in the Great Depression, the former, of- ciency through automation has led to man existence. Within a thirty-month ten called “the silent generation,” is job loss. And many of the new jobs cre- period, enough people are added to the small comparatively. It escaped World ated pay lower wages than those lost in world to populate another U.S., with War II, experienced “the forgotten manufacturing. Dear says the 1970s ninety-five percent of this world popu- war” in Korea, and entered the work- and 1980s were characterized by low lation growth occurring in developing force during a period characterized by growth in labor productivity, company countries. high levels of education, low unem- downsizing, plant closures, extensive ployment and high socio-economic op- loss of the better-paying manufacturing Despite claims that natural resources portunity. This generation, particularly and factory jobs, sharp reductions in remain plentiful and technology can middle-aged women, is currently caring agricultural labor, costly health care, solve environmental problems, Dear for aging parents who are living into high inflation, and increasing taxes on contends that resource distribution poli- their 80s and 90s. Furthermore, the ra- the middle class. These factors, he pro- cies will become a growing political tio of care givers to care receivers is in- poses, seriously stagnated the incomes challenge. He agrees with Paul creasing, Dear says, as there are pro- of middle and lower class Americans. Kennedy’s thesis in Preparing for the portionately fewer people age 50-64 to Unfortunately, these trends continue. Twenty-First Century that world popula- care for those 85 and older. tion cannot be supported if current consumption attitudes, patterns and The third layer of the “population Technological Change levels continue. sandwich” is the 80 million people age Many jobs, Dear points out, now re- 12 to 32. Numerically larger than the quire computer literacy. He quotes Pe- In this country, although the popula- “baby boomers,” they tend to see ter Drucker’s prediction in Post-Capital- tion is growing at a slower rate than in themselves as maturing just as the soci- ist Society that “the basic economic developing countries, it is still faster ety seems to be deteriorating, with up- resource...is no longer capital, nor nat- than in the European Union, largely ward mobility opportunities apparently ural resources...nor labor. It due to immigration. Minority popula- decreasing.This disenchanted genera- is...knowledge.” The advent of the in- tions are growing more rapidly than the tion is the most diverse in terms of formation era has changed the nature of white population. Acknowledging that backgrounds and orientations, Dear jobs. we share problems with developing says, and the most tolerant of new def- countries in terms of income inequality, initions of life styles, social norms and disparities in health care, high unem- Education family forms emerging out of the civil ployment and dislocation, and social di- rights, anti-war, and women’s move- Dear believes the U.S. is moving to- vision, he also refers to our uniqueness. ments and the sexual revolution. The ward two separate sub-societies, one Although the U.S. sustains less than social, economic and cultural differenc- consisting of well-educated and trained five percent of the world’s population, es associated with the different eras ex- professional, technical and managerial it consumes a dramatically dispropor- perienced by the three major segments workers; another made up of less well- tionate amount of its resources. of the U.S. population have created un- educated and less skilled low-paid Dear also gives attention to the age derstandable strains in the develop- workers. He refers to former Secretary structure of the U.S. population. A pop- ment of social welfare policy and pro- of Labor Robert Reich’s pronouncement ulation increase of 125-150 million new grams that are responsive to and that 20% of the U.S. population com- residents in the next fifty years will, he acceptable to all three groups. petes successfully socio-economically suggests realistically, increase the need while 80% finds itself falling behind. A parallel development analyzed by for new infrastructure development, Education is the key to economic ad- Dear is the change in the racial and eth-

Evaluation Forum n Issue 12 n Summer 1997 3 Features

nic composition of the population. In est one-fifth of all households fell by women and minorities in many occupa- 1992, 75% of the U.S. population was 10%. At the same time, the income of tional niches. classified white, 13% black, 9% His- the top fifth increased by 34%. Accord- panic, and 3% Asian Pacific Islanders. ing to the Center on Budget and Policy Waning Faith in Government The proportion of ethnic minorities has Priorities, the gain for the top level was gradually increased in the 1980s, as has a staggering 122%! Wages and salaries The U.S. was established by people the annual growth rate. The annual for this well-educated, highly skilled who left oppressive authoritarian gov- growth rate for whites was 1% but for group more than doubled from 1977 to ernments. Consequently the Constitu- minorities it was much higher — 1.5% 1988. Dear also alerts us to tax breaks tion incorporated a formal system of for blacks, 2.2% for Native Americans, for this group and the increase in capi- checks and balances to prevent the for- 6.4% for Hispanics, and 6.62% for tal gain income. At the other end of the mation of a powerful and intrusive gov- Asian Americans. The projection for income continuum, AFDC benefits de- ernment. Unusual emphasis was placed 2050, given rates of growth in the clined 45% from 1970 to 1993. The dis- on the rights of states and individuals. 1990s, is that whites will constitute parity between high and low income But this diffusion of power, Dear sug- 53% of the population, Hispanics more populations had become striking and gests, has slowed the development of than 20%, blacks 16%, and Asian disturbing. Furthermore, according to social welfare policies and programs. Americans 11%. This will create a more data available in 1966, trends toward And other qualities of government have ethnically diverse population, but likely income inequality are continuing into been barriers in responding to social void of a commitment to the “melting the 1990s. need. Deficits, special interest group in- pot” concept that distinguished our ear- fluences, and shrinking revenues for Beyond these figures, Dear tells us lier blending of historical roots, religions new social expenditures have paralyzed that the number of Americans in pover- and cultures into a single-language, sin- government action, precluding new initi- ty climbed in 1992 for the third consec- gle-culture society. atives, and an evaluation of the con- utive year, higher than in any year tinuing effectiveness of programs as a since 1962. Approximately one in seven basis for positive change. Changes in American Family Americans were classified poor in 1992. Norms, Structures and Life Styles Slightly over 30% of blacks, and close The Impact of the Trends Dear joins other experts in viewing to 30% of Hispanics were determined Dear concedes that some programs the dramatic rise in the number of sin- to be poor, while the poverty rate was for the poor have received more fund- gle-parent families as a major legacy of 12% for whites and the same for Asian ing in the 1980s and 1990s, but he changing family patterns, linked inevi- Americans. claims there is ample evidence that tably with rising poverty. In 1970, 11% In 1992, one-fourth of children under most anti-poverty efforts have not ex- of all families were headed by a single age six were poor. Close to 22% of youth panded consistent with economic parent, usually a woman. By 1990, under 18 were poor. Nearly half of all growth, inflation and human needs. nearly a quarter of all families were so black children under 18 were poor. And Not only income assistance but also structured. Dear looks for causes in the there was an increase in the poverty rate employment and training and housing higher divorce rates, the lower propor- for black married couples, which is sig- programs have been reduced, while tax tion of remarriages, and the significant nificant in the context of poverty rates benefits for upper income people have increase in never-married mothers. By typically being much lower for couples grown substantially. The author is par- 1993, there was one divorce for every than for single heads of households. ticularly interested in the effects of spe- two marriages, on average. The number One-half of all female-headed families cial income tax deductions, exclusions, of young unmarried men and women with children were poor. The absolute preferential tax rates, credits and defer- had risen -- by 1990, 63% of women level of poverty was also shocking. Al- rals of tax liabilities, that represent sub- age 20-24 had never married, compared most one-half of all American children stantial benefits to those with sufficient with 28% in 1960; 79% of men in this in poverty lived in households with an income to take advantage of such poli- age group had never married, compared income less than half of the official pov- cies. He reports that in 1994, tax bene- with 53% in 1960. In 1960, one in erty level! Furthermore, these trends were fits of these kinds cost the federal gov- twenty births was to an unmarried in marked contrast to the lowering of ernment $256.4 billion. They are mother; in 1990 more than one in four the poverty rate for the elderly. projected to cost $337 billion by 1998, births, and almost two-thirds of all considerably more than the projected black children were born to unmarried Dear calls attention to the fact that federal deficit for that year. mothers. information about these demographic trends has dispelled the notion that un- Perceived economic scarcity, tax Dear also comments on the composi- employment is always the primary cor- welfare for the affluent, and the desire tion of families, indicating the increase relate of poverty. Approximately 40% of to shrink government and limit its con- in reconstituted families and unmarried the poor were employed in 1992. Ten trol has spawned a new “linguistic sys- but co-habiting couples. He emphasiz- percent had full-time, year-round jobs. tem,” — devolution, decentralization, es changing family work patterns, with A changing economy, moving rapidly deinstitutionalization, declassification, both spouses in 60% of two-parent from manufacturing to services and in- reconfiguration, consolidation, privati- families in the workforce in 1990, a sig- formation, has eroded high-paying low- zation, volunteerism. The broader “wel- nificant change from 40% in 1970. er-skill employment, and increased fare state” view, in which benefits and competition for emerging high-income- services enhancing life quality and op- Change in the Distribution of generating jobs. And racism and sexism portunity are seen as rights of citizen- Income and Wealth continue to limit the educational, train- ship, was lost, Dear concludes. The Dear shows that between 1977 and ing and employment opportunities of general public had failed to under- 1988, the after-tax income of the poor-

4 Evaluation Forum n Issue 12 n Summer 1997 Features

stand, or had preferred not to acknowl- suggesting policy revisions. The conclu- nn edge, he believes, that almost all major sion of his chapter is bold, his proscrip- A Generational social programs and policy victories had tion controversial in the present politi- been accomplished at the federal level, cal environment (see Figure 1). Perspective on often in opposition to less universalistic Young Adults approaches at the state and local level. Relationship of Dear’s Analysis Neither did the public realize that the to Youth Issues administration of these large programs, In the previous article, Ronald Dear such as Social Security, were costing no The economic, social, cultural and po- litical trends Dear identifies as key in- provided an overview of recent demo- more in 1992 than 1% of the benefit graphic changes affecting the formation payments they distributed, a record fluences on social policy in the U.S. have an impact on government social of social policy. He emphasized the cul- even by corporate standards, Dear pro- tural divergence across four important poses. expenditures for youth education, train- ing and employment. Young people’s generations represented in the current There are reasons for optimism, Dear ability to acquire a level of knowledge American population: the elderly, the insists, that national social welfare and skills that affords choice and op- middle-aged, the Baby Boomers, and guarantees will not be totally disman- portunity in adulthood is critical to their the young adult children of the tled in the 1990s, but he sees a number future life quality, and that of their Boomers’ cohort. In this article we ex- of opposing forces at work which are families and their own children. The plore further some of the differences influencing and will likely continue to trends of concern to Dear have created between those maturing during and direct social policy development away vastly different societal environments shortly following World War II and the from the “welfare state” concept. And and socialization resources for different young adults of the 1980s and 1990s. he is concerned about the constraints groups of youth in American society, the trends he has analyzed place on substantially reducing life chances for Qualifications to the Generational progressive social planning: an increasingly large proportion of the Comparison youth population. If greater equity is n The decline in liberalism. The commentaries summarized here not achieved, Dear feels the disparities focus on the student generations of the n The growing national debt and con- in opportunity will have serious reper- tinuing high deficits. late 1940s-early 1950s, and the late cussions for American society in the fu- 1980s-early 1990s at a single liberal arts ture in terms of disenchantment, isola- n Social program cutbacks due to inad- college, Oberlin College in northern equate state and local revenue. tion and alienation. His fears are well Ohio. Long known for its high academ- worth addressing. ic standards and social consciousness, n Growing citizen distrust of govern- ment. Oberlin students have been selected traditionally not only for their academic n A changing economic base. Editor’s Note prowess but for their leadership quali- ties and their commitment to social and n Rapid movement toward internation- For readers interested in more de- al trading blocks. democratic values. The insights emerg- tail, please see “Social Welfare Policy” ing in a comparison of these two co- in the Encyclopedia of Social Work, horts of Oberlin students make an im- The Need for a Re-Invigorated 19th Edition. Washington, DC: NASW portant contribution to understanding Social Agenda Press, 1995. the changes that have taken place in Dear is not reticent when it comes to society over the decades involved, and in the environments in which youth grow to maturity in the 1990s. Howev- Figure 1 n Ronald Dear’s Reworked Social Welfare Agenda er, the uniqueness of this single case study requires caution in generalizing n The national political agenda must again be infused with a concern for poverty, hun- the commentary to other American col- ger, homelessness, unemployment, crime, and urban decay. lege students and to non-college-bound n Disparities in life quality between the more and less affluent must not be ignored. American youth. n Social programs must be updated and modified based on evaluations of their effec- The limited focus on Oberlin stu- tiveness, and new initiatives tested. dents is clearly outweighed by the n Program changes should be made incrementally rather than through radical surgery. knowledge and experience of the two n The tax system must be made more progressive. individuals making the comparison -- Nancy Dye, the President of Oberlin n Defense expenditures need to be reduced. College and an historian, and Oberlin n The level of Social Security benefits may need to be more consistent with the level of Professor Emeritus Albert McQueen, a contributions to the system. sociologist. This article draws ideas n The U.S. needs to develop an annual or biennial broad-based “social indicators” re- from their presentations on generational port. differences in the two cohorts of stu- n The U.S. should recognize and capitalize on its diversity without allowing special inter- dents at the 1995 reunion of the classes ests to fragment the society’s consensus on social welfare issues. for 1951–1953. The speeches constitute n Social policy reforms need to give more attention to basic social class divisions, em- a penetrating qualitative study that dis- phasizing solutions to systemic problems. tills major socio-cultural themes emerg- n Social policy must give more attention to increasing collective life quality, rather than ing in American society and their im- life quality only for certain groups.

Evaluation Forum n Issue 12 n Summer 1997 5 Features

pact on the values, attitudes and be- editorial board of the Journal of Ameri- unique societal influences at particular havior of two young adult generations can History. She has authored a book historical moments can have a decisive separated by a span of forty-five years. on feminism and the labor movement, impact on the efforts of young people These were the critical years following co-edited a book on gender, class and to cope with developmental tasks and World War II which witnessed rapid ethnic issues, and written a number of on the results of such efforts. For com- and far-reaching economic, social and book chapters, articles and papers on parative purposes, McQueen concen- cultural change in American society. As women’s history topics, including is- trates on two cohorts of students during a valuable illustrative case, these com- sues dealing with work, health, race, the distinctly different historical periods mentaries increase our understanding class and femininism. of the late 1940s to early 1950s, and the of the base of societal leadership that late 1980s to early 1990s. The following overview is based on helped form the social policies of the Dye’s and McQueen’s presentations at past three decades, and will shape its a 1995 reunion program titled “Under- Avoiding Overgeneralizations immediate future. standing Student Generations in a About Intergenerational Changing World.” Differences The Analysts As a researcher, McQueen alerts us Majoring in psychology, Albert Mc- Professor McQueen’s Analysis to the dangers of stereotyping college Queen was one of a small number of McQueen focuses his analysis on student cohorts as “conformist and si- African Americans graduating from three sets of developmental tasks that lent” (the phrase often used to de- Oberlin in 1952. Returning to teach at pose the greatest challenges to college scribe the generation of the fifties), as Oberlin in 1966, he became a member students, and the historical contexts “rebellious” (the sixties’ epithet), and of the joint sociology/anthropology fac- that shape youth’s perspectives and life as “conservative and materialistic” (the ulty and served as its chair for an ex- options. The three most important sets frequent assessment of the 80s’ genera- tensive period prior to his retirement in of developmental tasks facing students tion). Although these kinds of labels 1995. He was one of the earliest black are these: may contain important elements of faculty members at Oberlin, Oberlin truth, he cautions his audience not to having been the first liberal arts college 1. Developing and consolidating a satis- become entrapped in such broad gener- to admit African Americans and women factory sense of identity and com- alizations. on a regular basis. At the University of mitments to a set of values and goals Michigan in the early sixties, McQueen for one’s life. McQueen points to Paul Loeb’s stud- received a Ford Foundation Fellowship ies of American college students be- 2. Pursuing a liberal education: to pursue African Studies at the Univer- tween 1987 and 1993 reported in Gener- ations at the Crossroads. Loeb found sity of London, and a Research Fellow- n To acquire knowledge, skills and ship at the University of Ibadan in Ni- orientations that expand one’s in- that the majority of the students in geria to conduct research on selected tellectual horizons to include these studies, who were confronting youth problems in Nigeria and Kenya. many fields of interest. personal insecurities rooted in precari- A grant from the Social Science Re- ous socio-economic conditions and un- n To learn to think independently search Council supported the analysis certain futures, tended to be politically and critically. of his cross-cultural data. McQueen has withdrawn and somewhat apathetic. Consequently he labeled them adapt- served as President of the Association n To creatively integrate knowledge of Black Sociologists, and has held a from disparate fields. ers. Loeb also identified a significant number of positions in the American minority of students as activists. These n To gain specialized knowledge Sociological Association. He has a con- were the students who took public and skills useful in making career tinuing interest in the processes in- stands and engaged in actions intended choices. volved in youth socialization and their to bring about desired social changes. incorporation within the social struc- n To become sensitized to the moral tures of different societies and historical and ethical aspects of knowledge The Influence of Societal Change periods, with a particular interest in mi- and its uses as a basis for enlight- McQueen highlights certain key soci- nority youth. ened citizenship. etal changes in American society fol- Nancy Dye majored in history at Vas- 3. Developing self-confidence and a lowing World War II that have influ- sar College and received a doctorate in sense of belonging, by being recog- enced the ways in which college history from the University of Wisconsin nized and respected as a valued per- students have dealt with essential de- at Madison. She was a history professor son by significant individuals and velopmental rites of passage: at the University of Kentucky and Dean groups. n The economic boom of the late of Arts and Sciences at Vassar College McQueen sees the roots of early 1940s and early 1950s. before becoming Oberlin’s thirteenth identity and commitments in one’s so- and first woman president. Dye re- n Expanded job opportunities and in- cialization prior to young adulthood — ceived the Outstanding Professor Award creased socio-economic upward mo- in family, peers, the educational sys- from the University of Kentucky, as bility. tem, churches, and the immediate and well as fellowships from the Rockefeller larger communities. College, he says, n Suburbanization and the growth of and Ford Foundations, and a grant from serves to concentrate and speed the ac- the middle class. the National Endowment for the Hu- quisition of self-knowledge and a manities. She has served on numerous n Increased geographic mobility via ex- knowledge of the worlds of which one education-related boards, including the panded transportation systems. is an integral part. His view is that

6 Evaluation Forum n Issue 12 n Summer 1997 Features

n Expanded communications, with the formation of every kind via constant- nus and professor of history, Geoffrey development of electronic informa- ly developing electronic technologies. Blodgett: tion production technologies. Despite indications of social progress, We had the enormous good luck to McQueen points out that these changes McQueen believes these forces have believe that we possessed a desir- in the decade after World War II did created substantial anxiety and uncer- able future. The society we were not mirror substantial changes in the tainty for college students as they seek getting ready to enter seemed, on basic underpinnings of the society — resolutions to essential developmental balance, to be a good society. Most of us were eager to find our places that is, in traditional family and com- tasks. Added to this is the fact that the in its structure and fulfill ourselves munity values, attitudes and behavior. student movements of the 1960s and in ways expected of us. We were He reminds us, however, that deeply- 1970s undermined the guiding and remarkably career-oriented and rooted, oppressive prejudices and pat- mentoring roles of many adult authori- achievement-oriented. We thought terns of discrimination persisted, and ties (including parents), leaving an ab- we were needed by society, and by that the era of the Cold War and the sence of significant guide-posts in the the professions we wanted to pur- Korean War produced “highly virulent lives of many young people, who chose sue. The jobs were out there wait- pressures for ideological conformity.” independence and self-regulation as ing for us. We accepted the ethic of they contended with the challenges of apprenticeship in order to get In this era there was minimal ambi- ready for them. We entered train- growth and development. guity at Oberlin about what constituted ing programs and graduate appropriate academic standards and in- One expression of this new indepen- schools to find out what was ex- dividual behavior. The Oberlin heritage, dence is what McQueen calls “the pow- pected of us, required of us, and built on Christian doctrines and values, erful identity politics” of the many spe- we did it. has at its core principles of socio-politi- cial interest groups that have emerged This kind of attitude, Dye says, does cal and cultural progressivism. The au- on college campuses. In contrast to the not describe the world view of college thority of the administration and faculty earlier emphasis on “melting pot” ideas, students in the ’90s. as carriers of this heritage was rarely these groups are inclined to concentrate tested. Nevertheless there was diversity on the pursuit of their own interests Influences Shaping the within the student body. Many could and rights, which they may define in be classified as adapters, a large pro- terms of political, ethnic, racial, reli- Perspectives of Youth in the portion as activists, and a small number gious, cultural or other characteristics. 1990s as rebels (who, for example, struggled This has led to pervasive fragmentation Dye emphasizes several factors in for and accomplished the establishment in campus life.While such widespread seeking to explain some of the differ- of radical new living arrangements in preoccupations with special interest ences in orientation between post- student cooperatives.) group goals surely yield some positive World War II students and students to- benefits for those involved, McQueen day: The late 1980s and early 1990s was a feels that the general trade-off has been time of dramatic changes that contrast- “deep uncertainties and insecurities ed sharply with those of the earlier pe- Differences in Cultural among college students, widespread riod. Among the forces McQueen be- Realities conflicts in student life, and an atten- lieves have played a large part in dant loss of a sense of community.” A common “normative culture,” Dye shaping the life conditions, perspec- However, he sees important counter says, characterized the 1940s and tives and future prospects of college tendencies, in that many students who 1950s, pulling the society together students of this era are the following: are involved in identity politics also seek around a singular set of values, mores and attitudes about the privileges and n Economic and social stresses result- to cultivate common interests on cam- ing from a widening gap between pus and to serve the larger community responsibilities of citizenship, which the rich and the poor; growing glo- for the common good. was associated with a patriotic nation- bal interdependence and competi- alism. A relatively common view of country and society supported the cen- tion; increasing federal deficits and President Dye’s Analysis policy changes; increasing costs of tral role of political institutions and citi- educational and medical services; Dye proposes that the characteristics zens’ acceptance of them — an era she and continuing degradation of the of Oberlin students in 1995 are not dis- titles “middle class nationalism.” This environment. similar to those of previous eras in these middle class, rising out of the prosperi- respects: ty in the trade and manufacturing sec- n Vast expansion of ever-changing me- tors, represented a strong nationalistic n A passionate interest in the world. dia-promoted consumption patterns base that at the same time provided a and lifestyles. n The desire to explore this world intel- “sense of place” and “historical conti- lectually, socially and aesthetically. nuity.” It helped prevent a deterioration n Significant erosion of many tradition- of American society into contending al family and community values and n A strong commitment to achieve at a factions, and promoted a common feel- arrangements. high academic level. ing of community attachment and obli- n Growth of major social problems n A deep commitment to social gation. such as homelessness, drug use and progress. abuse, ethnic and gender conflict, The society of the 1990s, she con- violence, and AIDS. However, she also sees major differenc- tends, is more difficult to understand, es between current students and those to enter, and to accept. Traditional con- n Dramatic increases in the production of previous eras. She illustrates some of cepts of nation and citizenship, and of and near-instant dissemination of in- these with a letter from a fifties’ alum- institutional authority, are being ques-

Evaluation Forum n Issue 12 n Summer 1997 7 Features

tioned. Diversity is challenging our un- goal — the lowest percentage in the derstanding of community also. history of the survey. The percentage Editor’s Note who said they discuss politics frequent- A November 30, 1995 article in the Breakdown of the Nation-State ly also reached its lowest point. Dye surmises that specialized identity New York Times, “Boomers: ‘Not As I In the post-World War II years, groups have replaced some of the Did’ Parents,” provides interesting an- young people grew up with a loyalty to broader political commitment that was ecdotal support for one of the themes country and its institutions, and a de- such a strong quality of Oberlin stu- in the Oberlin presentations, namely sire to work for change within that sys- dents in the past. the legacy of too early self-indepen- tem. Again quoting from Blodgett’s let- dence in a more complex society and ter: As a result of these and associated world. The journalist, Trip Gabriel, trends, such as increasing economic in- claims that surveys suggest that the In courses in American history and equality, Dye feels that college students foreign policy, we began to absorb Baby Boom generation (now age 31- the new realist concepts of national today, and to large extent young adults 49) has a strong desire to protect chil- self-interest and controlled conflict generally, lack a belief that they will dren from a world that has grown more which underwrote the Cold War have the future they desire. They do threatening, and from an era in which containment policy and displaced not feel assured that they will be “bet- they believe they were afforded too older Wilsonian concepts of inter- ter off” than their parents. They are the much freedom too soon. These con- national law and moral idealism as first generation to harbor that anxiety. cerns about the future of children, proper guides to foreign affairs. In fact, many expect to be “worse off” about anomie and cynicism, cuts . . . In the arena of domestic Ameri- in terms of standard of living, with a across political and racial lines, he can history, “consensus” gradually new agnosticism about the future. It replaced “conflict” as the key says. Gabriel quotes from Generations, phrase for explaining how Ameri- may be secure, it may not be. They co-authored by historian Neil Howe, to cans normally thought and be- may marry and have children, they may make his point: not do so. They may have traditional haved toward one another. As we What Boomers do not want is for sexual orientations, they may not. They learned to locate the terms of their kids to grow up like Generation X American consensus in the past, feel more ambiguity and uncertainty did — no order, no structure, no inno- more often than not we also than before. learned to identify those terms as cence. Facing too many choices. Being values of our own. From political told the answers to questions they nev- science we absorbed comparable Changing Conceptions er had, at too early an age. Exposed to lessons. We discovered the intrica- of the Family too much. cies of the peculiar American party Close to 35% of Oberlin students Gabriel claims that the mantra of the system. . . . We learned how the now come from single-parent house- system worked, for all its idiosyn- Boomers was “question authority.” Now cracies, to master the tensions of a holds. Many come from blended fami- they are the authority. Surveys reveal a pluralistic society. Implicitly, and lies. Dye suggests that these students new more restrictive, rulemaking child sometimes explicitly, we received view relationships of all kinds as far rearing movement among them. A the message that it was better to more fragile and fluid than was the 1995 Roper survey found that good join a party and work within it for case in earlier eras. manners and politeness were at the improvement than to stand on the top of the list. Responsibility was sec- periphery wringing our hands in ond. Independence and imagination, dismay...We learned to our sur- Some Comments prise that most professors of politi- The analyses of disparate college high priority in their own era, were cal science frowned on political in- generations tell us important things much lower on the list. dependents as soft-minded, about the differences emerging in so- But some experts see this new inter- idealistic troublemakers. The inde- cialization patterns and attitudes relat- est in placing limitations on behavior, pendents’ notion that party politics ed to significant change in American and instilling values such as teamwork, spelled a degradation of pure rep- resentative democracy was, we society. Increasing uncertainty about as a product of both parents working were told, a self-fulfilling prophe- personal life choices and anxiety about and having less time to serve as role cy, because it removed them from one’s economic future are pervasive in models for their children. Many mea- effective involvement in the system the current youth generation, in the sures of adolescent well-being, for ex- they disliked. context of weakening ties with institu- ample, are declining: increasing mari- Dye proposes that rising economic tional authority and the political sys- juana and LSD use, a rising incidence internationalism and the decline of the tem. Clearly, these changing personal of sexual intercourse by age 15, a importance of the nation-state have re- perspectives and world views have im- greater potential for acquiring sexually duced students’ national loyalties, their plications for youth education, training transmitted diseases, increasing arrest view of the responsibilities of citizen- and employment policy. A prominent rates for violent crime, a growing media ship, and their interest in social activ- implication is that such policy must be- focus on sex and violence. Some youth ism within the political system. In a gin with a broader understanding of specialists do not see a turnaround, 1994 survey of incoming Oberlin fresh- the relationship between societal and Gabriel says, until the larger second man, only 31.9% said that “keeping up global economic, social and cultural wave of children of the Boomers (now with political affairs” was an important change and the achievement of generic 8 and under) enter adolescence. youth development tasks.

8 Evaluation Forum n Issue 12 n Summer 1997 Features

nn Interested readers may want to delve come of all American social classes rose into the following: Rising Income at nearly the same rate. The concept of “a rising tide lifting all boats” seemed Dye, N. (co-editor with Noralee Frankel) Inequality and to reflect real life, and indeed the pov- Gender, Class, Race and Reform in the erty rate decreased, by half between Progressive Era. Louisville, KY: Univer- Poverty sity of Kentucky Press, 1991. the late 1940s and early 1960s, and again by half by the early 1970s. Now Dye, N. “The Rise of the New Woman: Sheldon Danziger and Peter wages are much lower and fewer jobs Women’s Work, Women’s Politics, 1870- offer fringe benefits. Unemployment 1928,” in M. Cantor (editor) Main Prob- Gottschalk have collaborated on numer- lems in American History, Volume 2. New ous studies of poverty and unemploy- rates for both white-collar and blue-col- York, NY: Dorsey, 1988. ment, and have jointly published their lar workers are higher. Both adults in results in article and book form. This two-parent families must work to allow Dye, N. As Equals and As Sisters: Femi- for continuing economic support and nism, the Labor Movement, and the article reviews their most recent, very Women’s Trade Union League of New timely book, America Unequal, pub- home ownership. The retirement age York. Columbia, MO: University of Mis- lished by the Russell Sage Foundation has been raised, and social security souri Press, 1981. and Harvard University Press in 1995. benefits in the future will mirror the lower rate of return on social security Horowitz, H. L. Campus Life: Undergraduate Danziger is Professor of Social Work and Cultures From the End of the 18th Cen- Public Policy at the University of Michi- taxes, significantly affecting young tury to the Present. Chicago, IL: Univer- gan. Gottschalk is Professor of Econom- workers. sity of Chicago Press, 1987. ics at Boston College. These younger workers face a soci- Littwin, S. The Postponed Generation: Why The theme of the book is that the ety characterized by rising inequalities American Youth Are Growing Up Later. global and domestic economy, techno- in wages, income and wealth, and a New York, NY: William Morrow and Co. high rate of poverty, according to the 1986. logical change, family structure and public policies have contributed to the authors. Such conditions are the legacy Loeb, P. R. Generations at the Crossroads: gradual decline in family incomes, to of the 1980s and early 1990s, they say, Apathy and Action on the American rising economic inequality, and to an during which growth in income was Campus. Berkeley, CA: University of skewed toward the top of the income California Press, 1989. erosion of job opportunities for lower- skilled workers. Based on empirical distribution, and inequalities within Modell, J. Into One’s Own: From Youth to studies made possible by grants from and between demographic groups wid- Adulthood in the United States, 1920- the Russell Sage Foundation and the ened. Different groups began to experi- 1975. Berkeley, CA: University of Cali- ence disparate economic fortunes. The fornia Press, 1989. Presidential Initiatives Fund of the Uni- versity of Michigan, the authors reject slender gains went disproportionately to the notion that increasing poverty in the old, the better educated, two-per- the U.S. is the result of a weakening son families, and whites. work ethic, a legacy of social programs, or associated with the government’s The Inner-City Syndrome failure to shore up the middle class and The popular view has been that address the increase in poverty. Their these changed conditions reveal a lack recommendations fall heavily on de- of work effort and skills. The public mand-side as opposed to supply-side looks to inner city poverty, single moth- solutions. ers, and displaced blue collar workers as the primary victims. Danziger and A Departure from the Past Gottschalk say this is a distortion of re- The authors comment that the ine- ality. The new inequalities have affect- qualities in the America of the 1990s ed all population groups, even college represent a substantial departure from graduates. In 1991, for example, 16% our recent economic history. Family in- of male graduates and 26% of female comes doubled in the two decades fol- graduates between age 25 and 34 lowing World War II. They have grown worked at some time during the year only minimally since then. More strik- but earned less than the poverty line ing, many workers have lower real for a family of four. In 1973, this was earnings than in the 1970s. Youth and true for only 11% of the men and 37% young adults have assumed that they of the women. The demand for less- would experience the same mobility skilled, less-experienced workers has and affluence as their parents, but the decreased noticeably. It is gradual, authors claim this hope has dimmed in even though not visibly dramatic the new economic and technological changes in the economic environment, environment. the authors contend, that explain the changes in Americans’ standard of liv- The authors note that in the era of ing, not changes in the American work- the 1950s and 1960s, most families er. And the origins of the problem lie in were living out the “American the early 1970s. It is these structural dream” — good wages and fringe bene- economic changes that are the authors’ fits, house ownership, savings. The in- primary variables in the search for ex-

Evaluation Forum n Issue 12 n Summer 1997 9 Features

planations for rising inequality and pov- n Why didn’t the gains of the 1980s not exempt from this change. In the erty. “trickle down” more to the middle late 1980s and early 1990s, the share of class and the disadvantaged? income held by the middle class was The recovery in the 1980s looked like lower than in any other period since a period of renewed progress, but the n What changes have made more of World War II. Much of the growth in data hardly support this conclusion. the population rich without reducing the incomes of married-couple families This era did lead to job growth, declin- poverty? was due to the wages and work effort ing unemployment and inflation, and n What can be done to reduce poverty of wives, not to the rising wages of an increase in the gross national prod- and the economic inequalities that husbands. Meanwhile, the tax cuts of uct — but only a small portion of the have widened over the past two de- 1981 benefited the highest income fam- American population benefited. This cades. ilies, not the middle class. In fact, Dan- period “set the stage,” the authors say, ziger and Gottschalk see the 1980s as “for both a change in academic think- an “historical anomaly,” an era of rising ing about the level of distribution of liv- Answering the Questions family incomes and rising inequality. ing standards and a change in presi- The authors suggest that most re- The continued and unprecedented in- dential administrations. The issue of searchers did not see coming the crease in inequality during the recovery fairness moved from the background to change in average earnings. Neither did of the 1980s challenged the convention- center stage in the early 1990s.” Re- they envision the increase in earnings al wisdom about the effects of econom- search on inequality and poverty began inequality, the continuing high rates of ic cycles. to attract more attention. poverty, or the impact of the weakening of governmental concern and attention A “revisionist” view of the 1980s The Rich and the Poor emerged in the context of low wages to poverty and inequality in the 1980s. and benefits, worker displacement and In their analyses of wealth and pov- dislocation, and continued high unem- Family Income erty, the authors chose a conservative measure of poverty by using the official ployment rates for certain groups. And The authors point out that both me- the media began to notice the problems poverty line. In 1991 this meant $13,924 dian family income, the most common for a family of four. Since no compara- associated with corporate restructuring measure of standard of living, as well and downsizing, such as the unemploy- ble official definition of wealth existed, as mean family income grew almost they collected data on the top 5% and ment of young college graduates, col- continuously from 1949 through 1973. lege-educated middle managers, and the top quintile of families. Their ap- The public perception in the early proach was this: if the richest fifth of highly-paid blue collar workers. Consis- 1970s was that the U.S. was much rich- tent with this concern, the authors families is considered “rich,” and if er than earlier, and more affluent than their incomes remain the same while show that since the recession of 1973 other industrialized nations. Neither of to 1975 poverty rates rose more during the other four-fifths fall, the share of in- these visions accurately reflect current come possessed by the richest fifth will each recession than following each re- reality. covery, parallel with a declining general rise, resulting in an increase in inequal- standard of living and lowered consum- The 1973–1975 recession was the ity but not in the proportion of families er confidence. Poverty rates began to longest and steepest since the late who are rich. That is, the level and dis- move in a different direction than eco- 1930s. In 1980, the country was again tribution of income influence the pro- nomic growth. During the 1980s the top in recession, followed by a brief recov- portion of the population that is rich, end of the income distribution benefit- ery. At this point median family income but this segment does not increase un- ed much more than others. Their earn- was 7.4% below its 1979 value and 4% less the absolute real incomes of the ings and property income went up and below the 1973 level. For the first time rich increase. their federal tax rates went down. since the Great Depression, the “typical” family was worse off than it was a de- For example, if every family’s in- As data revealed an increasing gap cade earlier. The recovery beginning in come doubled, there would be no between rich and poor, public policy- 1982 was relatively strong and endur- change in income inequality, but makers could hardly avoid acknowledg- ing. Mean and median income grew. By poverty would decline and the ing the effects of the internationaliza- 1989 it finally rose above the previous percentage who were rich would tion of markets, technological high in 1979. But economic inequality increase. If, in contrast, the in- breakthroughs and structural economic and poverty kept increasing. The 1970s comes of most families stayed change, and began to seek solutions. and 1980s had essentially produced little constant but the incomes of those But proposing solutions was not easy. improvement in the median standard of near the top of the distribution Macro-economic approaches have been living. Adjustments for the shift toward increased, there would be no one focus, tax and social policies anoth- more female-headed families failed to change in poverty, but inequality er. To assist the reader in evaluating the explain this paradox. and the percentage of rich would alternatives, Danziger and Gottschalk increase. review the evidence and pose these What stands out in the authors’ questions: analysis of family income data is that The poverty rate fell substantially all of the increased inequality between during the rapid growth of the postwar n Why have poverty rates remained so 1982 and 1989 is explainable in terms years -- from approximately 40% of all high? of the income gains for the richest 5% persons in 1949, to 22% in 1959, to of American families. This top group re- 14% in 1969. In that same period, the n Why has the gap between the rich and poor, and between the rich and ceived more total income than the en- rich increased from less than 1% to the middle class grown so wide? tire bottom 40%. The middle class was more than 5%. The middle class also

10 Evaluation Forum n Issue 12 n Summer 1997 Features

grew rapidly, from 60% of all persons grams have provided less cash assis- creased work hours, particularly on the in 1949 to 80% in 1969. But although tance to low-income families in a period part of female spouses, has been the the poverty rate tended to rise and fall in which the economy has increased primary coping strategy. with the condition of the market, it be- the need for such assistance. In 1949 men’s earnings represented gan to increase more during the reces- The authors’ research confirms that 82.7% of family income. Between 1949 sions and to decline less during the re- the past twenty years have witnessed a and 1969, growth in men’s earnings ac- coveries. Danziger and Gottschalk do divergence of living standards among counted for 73% of the rapid growth in not imply that economic growth no various groups in American society, family income, even though the propor- longer mattered, but rather that the re- with some of those receiving the lowest tion of married mothers with earnings lationship between the poverty rate incomes experiencing the most stag- doubled. In 1991, men’s earnings still and economic forces was more complex nant rise in living standards and the accounted for 63.4% of total adjusted than thought. largest increases in poverty. Non-His- family income, the earnings of their If the economy were to stay on its panic whites are better off than blacks, wives representing 21.5%. Clearly the present course, the authors estimate Hispanics and other minorities. Noneld- largest source of income for these fami- that median family income would have erly adults are in a better position than lies remained the men’s earnings. After to grow to $39,000 (over 13%) to return female-headed families. Workers with the early 1970s, however, the growth to the 1973 poverty rate level. Their more education are faring better than rate of men’s earnings slowed, more point is that without dramatic changes those with less. Current poverty rates for wives entered the workforce, and wag- in underlying demographic and eco- minorities and persons in families head- es and hours of work decreased. By nomic factors, poverty is destined to ed by unmarried women are above 25%. 1991, two-thirds of all mothers in two- stay well above the 1973 level. To sup- Black, Hispanic and white children liv- parent families had earnings, but the port this conclusion, the authors calcu- ing in mother-only families have poverty absolute level of earnings of men had late the poverty rate using alternative rates as high as 40%. stagnated. measures. These lead to the same end Although men’s earnings remained result, or to an even higher poverty Children of Minorities the largest source of family income, they rate. The trends for minority children are accounted for only 2.5% of the growth At the beginning of the War on Pov- particularly worrisome. While children in family income between 1973 and erty, the reasonable assumption was benefited from the economic progress 1991. In the 1990s wives’ earnings actu- that poverty could be reduced by following World War II, they did less ally replaced those of men as the prima- healthy economic growth, through mac- well in the 1970s and 1980s. Significant ry reason for any growth in family in- roeconomic control of the business cy- to the economic status of minority chil- come. Wives’ earnings now accounted cle. The Reagan Administration, the au- dren, over the past two decades these for 95% of the increase in their chil- thors remind us, blamed the stagnation children have been more likely to live dren’s standard of living. of the 1970s on the rise in oil prices with one parent than with two. Follow- These results, based on careful sta- and on President Carter’s economic ing 1973, while the median income for tistical studies, again convinced Dan- management, and they assumed that a all African Americans increased by ziger and Gottschalk that it is adverse speed-up in the economy would be 16.4%, the median income for black economic changes — in particular the shared equally by all groups. None of children did not increase, and the me- declining earnings of men — that ex- these assumptions was correct. dian income for Hispanic children fell plain current economic hardship, rather by 11%. This greater divergence for than sheer demographic change. Mean Disadvantaged Groups children than for other persons is of male earnings have hardly grown over great policy concern. In 1991, the medi- A major change since World War II the past two decades, and inequality in an standard of living for white children is the shrinkage of that part of the pop- men’s earnings has increased. was 96% of all persons; for black and ulation, in each decade, that has lived Hispanic children it was 40% and 44% What the data show for white men in families headed by the nonelderly respectively. In 1991, the median black holds true for minority men. The grow- and the non-Hispanic white. Danziger or Hispanic child lived in a family ing inequality of the past two decades and Gottschalk caution us to review whose living standard approximated the had similar effects on the living stan- this trend in the context of the increas- standard for all Americans in 1949, only dards of white and minority men. How- ing proportion of the population that 1.24 times the poverty line. ever, since the median standard of liv- represents groups with lower-than-aver- ing of minorities has been much lower age living standards and higher-than- Men’s Incomes than that of whites, the inequality is average poverty rates — i.e. minorities greater. The authors are firm in reject- and female-headed families. They say Over the past twenty years, only the ing explanations of the disparity in the data indicate that increasing hard- elderly have experienced rising incomes terms of “race-specific behavior.” They ship in these groups cannot be attribut- and declines in poverty. This has been provide ample evidence that economic ed to “moral decline in the American due mainly to government spending. At change is the primary culprit. family” or the relative increase in the the same time the elderly’s safety net size of the minority population. In the was improving, economic changes were A case in point is educational bene- absence of demographic change, the unkind to working men. Men’s earnings fits. There has been a pattern of rising economy still would have created such increased substantially between the rates of low earnings for both white inequalities, and the hardships would late 1940s and early 1970s, and then and black male high school graduates, have been more severe if women had stopped — in fact, their real earnings with the two groups beginning to di- not gone to work in unprecedented were lower in 1991 than in 1973! In- verge in the 1970s. This has been true numbers. Furthermore, government pro-

Evaluation Forum n Issue 12 n Summer 1997 11 Features

for the earnings of college graduates as ty of incomes, particularly those of an increased number of workers are well. The authors contend that conser- men, as the major explanatory influenc- temporaries. The number of these vative explanations of the differences in es. workers grew sevenfold between 1972 earnings between these two racial and 1991. By 1992 temporary employ- There was much debate in the late groups have vastly overestimated the ment was as large as the steel and auto 1970s and early 1980s about whether role of individual motivation, family industries combined. And rather than increased inequality was correlated structure and detachment from the la- responding to business cycles, the rate more with long-term structural changes bor market. Although these factors of temporary employment has shown a in labor markets than with temporary have contributed to the disparity, the steady upward trend. responses to recessions. However, the authors say, they do not explain much authors point out that as economic of the increase in economic hardship growth expanded in the 1980s and The Author’s Conclusions over the past twenty years. earnings inequality also grew, it became The two researchers are quick to ac- obvious that the U.S. was experiencing knowledge that we are only beginning Female-Headed Families long-term economic changes. Full-time, to understand the combination of com- The public view has been that a year-round workers, not only labor mar- plex influences that result in the large proportion of female heads of ket groups with access to government growth of income inequality in the U.S. household are on welfare rather than in transfers, were sustaining large increas- and the significant interactive effects of employment. But the authors’ data sug- es in inequality. And earnings inequali- these factors. In summary, however, gest otherwise. Women’s earnings in ty for all men continued to increase as they find strong evidence regarding the employment are the most important reductions were made in unemploy- role of these influences: source of income for these families. ment insurance and welfare. n Increased globalization of markets. Their earnings are low, however, be- Technological as well as economic cause women’s wage rates are lower, changes have played a significant role n Improved efficiency, whose negative there are fewer adults to obtain em- in increasing inequality, according to side is greater inequity. ployment within the family, and these the authors. Computerization has in- n Technological change. families receive so little cash assistance creased the demand for higher-skilled from government programs. Declining workers with a resulting decrease in n Change in the structure of the labor government cash transfers have not the employment opportunities and wag- force. made much of a difference in their liv- es of those without computer skills. These changes have both positive and ing standards. By 1991 government The de-industrialization of the U.S. has negative effects on income inequality, cash transfers amounted to only 10.3% reduced earnings significantly, and up- but the overall trade-off has been in- of the mean adjusted income of fami- ward earnings mobility has not creased poverty. Government’s primary lies headed by women. They accounted changed appreciably over a long peri- obligation in a capitalist democracy, the for 20% in 1969. If these women were od. authors suggest, is to develop policies not working, the authors point out, that help offset the inequities resulting their living standards would be consid- Again, different experts have differ- from these changes. erably lower than they are now. In ent explanations concerning the role of 1973 cash transfers averaged .42 times technological change, some criticizing the poverty line and accounted for 32% the measures used to study its effects. An Anti-Poverty Strategy Based of the income of mother-only families. But the authors point out that the nega- on the Research By 1991 these transfers averaged only tive effects of such change on earnings Danziger and Gottschalk are clear .24 times the poverty line, or 17% of to- has been well documented not only in about the lessons we should have tal adjusted family income. general statistical studies but also nu- learned about public policy over the merous industry-specific studies. Also, past several decades. Their message is Increasing Poverty it is now fairly well established that that policies that are not consistent market forces that increase efficiency The authors give much of their at- with the public’s perspectives and ex- do so at the price of equity. Meanwhile, pectations will not be successful or be tention to myths vs. root causes. For ex- the public sector has been left to com- ample, they point to analysts who tend appropriately implemented. For exam- pensate for these negative effects of ple, the negative income tax concept to blame the increasing divorce rate economic and technological progress. It and rate of out-of-wedlock births for failed the public acceptance test. The chose not to rise to this occasion in the potential efficiency in welfare mothers’ the high poverty rate, insisting that the 1980s. data do not fully support this conclu- caring for their young children rather sion. Although the shift toward more Job loss is sometimes given as an ex- than working has failed also. Education female-headed families raised the pov- planation for increased inequality. How- and training programs have proven to erty rate by 1.6% over an eighteen-year ever, the authors report that the extent be initially expensive, even though in period, or by about .1% a year, they of job loss was about the same in 1979 some the costs have been offset by sig- say, the poverty-reducing effect of the and 1989, suggesting that this factor nificant benefits to recipients. In terms economic growth of the past two de- accounted for very little of the differ- of taxpayers’ tolerance for policies in- cades has been offset by the increased ences in earnings. These losses mainly volving substantial spending, some ex- inequality of family income. The au- reflect changes in the industrial struc- perts estimate it would take human cap- thors credit the much slower growth in ture of the labor force, according to ital investments of about 1.7 trillion (in living standards and the rising inequali- Danziger and Gottschalk. For example, 1986 dollars) to reduce inequalities suf- ficiently to bring the wage distribution up to the 1979 level!

12 Evaluation Forum n Issue 12 n Summer 1997 Features

Another lesson is that market forces Editor’s Note Readers are also encouraged to alone are not capable of resolving the read the National Commission for Em- economic hardship we are now experi- Those interested in other publica- ployment Policy’s On Shaky Ground: encing, and therefore the authors view Rising Fears About Incomes and Earn- government action as essential. In that tions of the authors may wish to delve into the following: ings by Steven Rose, published in context, they concentrate on an ade- 1994. Rose’s analysis supports the the- quate governmental response, particu- Peter Gottschalk sis in America Unequal. The conclu- larly assistance in improving the em- “Transfer Scenarios and Projections of Pov- sions in this monograph are these: ployment and earnings of those with erty into the 1980s,” in Journal of Human n The American economy is providing low skills, and those most adversely af- Resources 16, 1981. fected by economic change: upward mobility for a declining pro- Policy Changes and Growing Earnings Ine- portion of Americans. n For those with jobs: expanded wage quality in Seven Industrialized Nations. n supplements and child care. Research paper. Boston, MA: Boston Lack of access to post-secondary College, Department of Economics, education is a barrier to job oppor- n For those who want to work but 1994. tunities. cannot find regular employment: [Co-authored with S. H. Danziger] “Macro- n A widening income gap is generat- transitional public service jobs at economic Conditions, Income Transfers, the minimum wage. ing anxiety even among well-edu- and the Trend in Poverty,” in The Social cated and experienced workers. Contract Revisited: Aims and Outcomes n Expansion of the Earned Income Tax n Credit. of President Reagan’s Social Welfare A declining number of men are up- Policy edited by Lee Bawden, 1984. wardly mobile — in every income, n Refundable income tax credits for [Co-authored with S. McLanahan and G. age and education category. the working poor. Sandefur] “The Dynamics of Intergener- n ational Transmission of Poverty and Wel- Average male earnings over the en- n State income tax relief for the work- fare Participation,” in Confronting Pover- tire decade of the 1980s were 4% ing poor. ty: Prescriptions for Change edited by S. lower than the 10-year average of the 1970s. n Employer subsidies. H. Danziger, G. D. Sandefur, and D. H. Weinberg, 1994. n Women are earning more than men, n Public service employment. [Co-authored with G. Sandefur] The Growth due to longer work hours. n Child support. of Earnings Instability in the U.S. Labor Market. Brookings Papers on Economic n Higher family living standards are The focus is on earnings supplements Activity 2, 1994. depending on women’s earnings. and work opportunities, rather than n work incentives and requirements. The Young adults are much less suc- authors emphasize the demand side of Sheldon H. Danziger cessful now in advancing their ca- reers. the labor market, and apply their strat- [Co-authored with P. Gottschalk] “The Pov- egy to all those negatively impacted by erty of Losing Ground,” in Challenge n The relative economic standing of economic, technological and related May-June, 1985. black men has fallen dramatically. changes, not simply the poor, estimated “The Poor,” in Human Capital and America’s to be roughly 47% of family heads. It is The U.S. is viewed as being in the Future edited by D. Hornbeck and L. midst of a profound economic transi- an impressive empirically-based and Salamon. Baltimore, MD: Johns Hopkins ambitious agenda in a period of uncer- University Press, 1991. tion. Those with access to skills and technologies are doing relatively well. tainty and unprecedented government [Co-authored with S. Danziger] “Child Pov- retrenchment. But new skill and technical require- erty and Public Policy: Toward a Com- ments are polarizing the economic prehensive Antipoverty Agenda,” in Daedalus: America’s Childhood 122, well-being of those with the necessary 1993. skills and those without. [Edited with P. Gottschalk] Uneven Tides: The author asks for more flexibility Rising Inequality in America. New York, in the labor market, through portable NY: Russell Sage Foundation, 1993. health care and pensions, education [Edited with G. Sandefur and D. Weinberg] and training, counseling and job assis- Confronting Poverty: Prescriptions for tance, accurate labor market informa- Change. Cambridge, MA: Harvard Uni- tion, family support, and partnerships versity Press, 1994. between capital and labor. The push is for more adaptive labor market policies which help workers adjust to significant change.

Evaluation Forum n Issue 12 n Summer 1997 13 Features

nn their level of wealth can have quite dif- and 1983. It had almost doubled since A Victim of Policy ferent life quality. In periods of eco- 1962. Average financial wealth grew nomic stress due to unemployment, ill- faster than marketable wealth during Neglect? Wealth ness or family breakup, wealth is an 1983–1989, reversing the relationship of Inequality important cushion. And fortunes can the 1962–1983 period. Average house- be a source of economic power and so- hold income also grew faster in 1983– Poverty specialists focus most often cial influence over a number of genera- 1989 than in 1962–1983. But in both pe- on social and income inequality. Wealth tions. riods average income grew more slowly inequality is frequently ignored. How- than average wealth. The growth in av- ever, the latter issue is beginning to Contemporary Wealth Inequality erage wealth tended to mask changes capture analysts’ attention in the in its distribution. Wolff says we must 1990s — the extent of current dispari- The 1980s offered enormous benefits look at medians rather than means. The ties in wealth are unprecedented. The to the top of the wealth distribution, median of the wealth distribution is de- combined effects of income and wealth while the rest of the population experi- fined as the level of wealth that divides inequality have significant implications enced a decline. In fact, 80% of house- the population of households into two for American society, and therefore for holds experienced a decline between equal-sized groups. If only the top quin- its youth. 1983 and 1989, if one looks exclusively tile experiences an increase in growth, at real financial wealth, such as bank Consistent with the increasing inter- median wealth is unaffected. When accounts, and stocks and bonds. Even trends in the mean deviate from trends est in wealth inequality, we review Ed- if housing wealth is added, the bottom ward N. Wolff’s new book, Top Heavy: in the median, this tells us that gains 40% of households had less wealth in and losses are unevenly distributed. A Study of the Increasing Inequality of 1989 than in 1983. Wealth in America, published in 1995 During 1983–1989, mean wealth in- by The Twentieth Century Fund Press. During the same general period, the creased by 23% but median wealth Wolff is professor of economics at New racial distribution of wealth declined grew only by 8%, with the bulk of the York University. He is also managing ed- from an already low level. The relative gains concentrated at the top of the itor of the Review of Income and income of African Americans, Wolff distribution. The share of marketable Wealth, associate editor of the Journal says, held steady at approximately 60% net worth held by the top 1% rose to of Population Economics and Structural of white income in the 1980s, but the 39% in 1989, while the share held by Change, a council member of the Inter- relative wealth of most black families the bottom 80% fell by more than one- national Association for Research on In- deteriorated. Wolff sees historical differ- fifth. This trend is reflected in financial come and Wealth, and a member of the ences in wealth between whites and net worth, which was distributed even editorial board of Economics Systems blacks as a legacy of slavery, discrimi- more unequally than total household Research. His principal areas of research nation and low incomes — but this con- wealth. are productivity growth and the distri- dition grew worse between 1983 and bution of income and wealth. 1989. In 1983, the median white family In 1989, the top 1% of financially had eleven times the wealth of the me- wealthy families owned 48% of the to- dian nonwhite family. By 1989 this ratio tal. The top one-fifth accounted for A Perspective on Wealth had risen to twenty times. Now more 94% of total financial wealth, and the Inequality than one in three nonwhite households second one-fifth accounted for nearly Wolff confirms that most discussions have no positive wealth at all, com- all of the remainder. And the concen- of distributional issues have concentrat- pared with one in eight white house- tration of financial wealth increased to ed singularly on income, or the mea- holds. the same degree as marketable wealth. sure of a household’s economic position Furthermore, the top 1% of wealth from year to year. His position is that Historical Trends in Wealth holders received 62% of the total gain in marketable wealth between 1983 wealth is a better indicator of long-term Inequality economic security. Wealth is the sum of and 1989. The bottom 80% received the current value of all the assets a The 60-year high in wealth inequali- only 1%. This trend was a shocking household owns minus all the house- ty in the U.S. was 1989, with the top turnaround, Wolff says, from the 1962– hold’s liabilities. Wealth, as well as in- 1% of wealth holders controlling 39% 1983 period, when every group enjoyed come, can vary over time but the of total household wealth. The richest some share of the overall growth in former, Wolff insists, is the foundation 1% owned 48% of the total. However, wealth, and gains were roughly in pro- for a family’s long-run security. the rise in wealth inequality from 1983– portion to the share of wealth held in 1989 was unusually dramatic. The 1962. Although certain kinds of income share of the top 1% of wealth holders come from wealth, such as interest on rose by 5%. The wealth of the bottom Income distribution also became savings or mutual fund accounts or 40% declined. Almost all of the abso- more concentrated between 1983 and rent, Wolff establishes that income and lute gains in real wealth were absorbed 1989. Most of the relative income gain wealth are not identical. Some kinds of by the top 20% of wealth holders. went to the top 1%, whose share of to- income do not flow from household tal household income grew by 4%. Al- wealth and many forms of wealth do American real wealth grew between most all the loss in income was felt by not produce income. Although both are 1962 and 1983, growing even faster af- the bottom 80% of the income distribu- sources of well-being, wealth is the ter that. American marketable wealth tion. critical resource. Families differing in actually grew twice as fast between their income and families differing in 1983 and 1989 as it did between 1962

14 Evaluation Forum n Issue 12 n Summer 1997 Features

Changes in the Structure of 1962 and 1983, increasing to 16.5% in U.S. and United Kingdom, and a so- Household Wealth 1989. Owner-occupied housing was the cialist government in Sweden. This most important asset in 1962, 1983 and leads Wolff to conclude that public Wolff focuses his analysis of struc- 1989, but its gross value was no more policies in and of themselves likely ture on demographic differences, and than one-third of total assets, or its net do not explain similarities in the dis- differences in the composition of value more than one-fourth. In 1989, tribution of wealth. wealth. Households in the age group housing represented 29% of the gross Wolff made a separate study of 45–69 are now the wealthiest group in value of assets, and net equity in own- France to compare the distribution of the U.S., with those 70 and over the er-occupied housing was only 20% of wealth between that country and the next wealthiest, and households under gross assets. Business equity and other U.S., finding that the shares of the top 45 the next. The average wealth of real estate, and stocks, bonds and 1%, 5%, and 20% were significantly white and nonwhite households contin- trusts exceeded gross house value as higher in the U.S., indicating that ued to converge, but the gap in aver- important assets. age wealth between whites and non- wealth inequality is greater. This ap- whites remained very large compared peared to be related to the fact that with differences in income, and the Other Societies French households retain a much high- lower half of the wealth distribution for Wolff’s analysis of data from Europe er proportion of their wealth in the nonwhite households actually dropped is unsettling in terms of our vision of form of owner-occupied housing, which behind the lower half for white house- American society as the primary model is more equally distributed across the holds — strongly suggesting that wealth of social democracy. Although measures population than other assets. inequality for nonwhites has grown of household wealth and associated in- faster than it has for the general popu- dicators of wealth inequality differ Income vs. Wealth Inequality across nations, Wolff says the evidence lation. Wolff wants us to understand that suggests that in the early part of the wealth inequality in the U.S. has al- Wolff says that age is clearly a pre- 20th century wealth inequality in the ways been greater than income ine- dictor of wealth inequality in the U.S. U.S. was comparable to that in Swe- quality. The top 1% of wealth holders In 1983, the average marketable wealth den, but lower than in the United King- has typically owned one-fourth of total of families under age 45 was far below dom and the rest of Europe where the household wealth, compared with the the overall average — that of families upper classes traditionally controlled 8% to 9% of share of income held by over 45 was well above the average. most of the wealth. But by the late those at the low end of the income dis- And there have been important shifts 1980s this situation had been reversed. tribution. Thirty-seven percent of the in relative wealth across these groups. The U.S. now has a much higher con- total real income gain between 1983 Middle-age households gained at the centration of wealth than the European and 1989 was realized by the top 1% expense of younger and older house- Union. holds. Racial differences are significant of income recipients, compared to these as well. Most of the gains in wealth for Wolff focuses his analysis on the recipients receiving 62% of the market- nonwhite households between 1983 United Kingdom and Sweden, because able wealth gain; 39% of the income and 1989 were experienced by the up- of the greater accuracy, comparability gain went to the next 19% of income per half of the distribution rather than and availability of their data. What he receivers, and 24% went to the bottom those less affluent. Since 1970, for ex- finds is this: 80%, compared with income receivers receiving only 1% of the marketable ample, there has been no increase in A dramatic decline in the degree of n wealth gain. Eventually the top fifth of the rate of home ownership among individual wealth inequality in the the American population received more nonwhite families. While the gap in United Kingdom between 1923 and than 75% of the total increase in in- mean wealth between whites and non- 1974, with only minimal change come, and all of the increase in wealth! whites closed further during the 1980s, from then on. The share of the top The striking nature of these data reveal the gap in median wealth widened, re- 1% of wealth holders fell from 59% a worrisome chasm between social flecting a greater inequality in wealth to 20% in that period; by 1990 it classes in American society. for nonwhites. For the median non- was 18%. white family, the differential is large Historically, the distribution of The same trend in Sweden, although and growing. n wealth and the distribution of income the decline in wealth inequality was have been in parallel. Income inequali- An analysis of the composition of more continuous; by 1990 the share ty declined steadily during the Great household wealth tells us about the of the top percentile was 21%. ways in which households save. Be- Depression years and fell dramatically tween 1962 and 1989, there was a ma- n A reduction in wealth concentration during World War II. A slight decline jor change, Wolff says, in the saving in the U.S., United Kingdom and occurred between 1945 and 1953, fol- strategies of households. Savings from Sweden until the late 1970s — but lowed by a sharp rise between 1981 assets and equities declined from 52% during the 1980s a sharp increase in and 1989. In 1989 both the concentra- of gross wealth to 38%, and savings the concentration of wealth in the tion of wealth and income inequality from real estate and unincorporated U.S. and Sweden compared to the flat peaked, the increase in both set a pre- business equity rose from 48% to 59%. trend in the United Kingdom. This cedent, with the exception of the Household debt as a proportion of net occurred in the context of conserva- 1920s. Poorer households had seen worth fell from 16.4% to 15% between tive economic policies in both the their net worth decline at the same time their incomes were deteriorating.

Evaluation Forum n Issue 12 n Summer 1997 15 Features

Interestingly enough, where there tions about the probable effects of have been discrepancies between wealth taxation, particularly revenue Editor’s Note wealth and income inequality, Wolff incidence and distributional effects. finds a cause in the ratio of stock prices Regarding the first concern, he finds Readers interested in probing Edward to housing prices. Stocks accrue prima- the Swiss version of a wealth tax the Wolff’s writings for more insights, rily to the upper class, housing to the most effective in raising funds. With re- please see: middle class. When stock prices in- spect to the second, since income and Growth, Accumulation and Unproductive crease relative to housing prices, the wealth are not highly correlated, he Activity: An Analysis of the Post-War share of wealth owned by the most af- sees a new wealth tax as shifting the U.S. Economy, Cambridge University fluent groups increases. Wolff contends tax burden away from young house- Press, 1987. that the data confirm this explanation. holds and toward more elderly ones, As co-author, editor, or co-editor: and from minorities to whites — again, Productivity and American Leadership: The The Taxation of Wealth those having benefited most and most Long View, MIT Press, 1989. able to contribute. His position is that In the U.S., household wealth is wealth taxes have important equalizing The Information Economy: The Implications taxed by way of capital gains taxes and effects on the distribution of income, of Unbalanced Growth, The Institute for Research on Public Policy, 1989. estate taxes. Individuals are exempted and these effects increase with age, are from taxation on the first $600,000 of greater for married couples than for sin- International Comparisons of the Distribu- wealth, and taxed at marginal rates be- gles, and are stronger among white than tion of Household Wealth, Oxford Uni- yond that exemption. Capital gains, nonwhite families. versity Press, 1987. which reflect the difference between International Perspectives on Profitability the selling price of an asset and its pur- The author reiterates that “virtually and Accumulation, Edward Elgar Pub- chase price when an asset is actually all the growth in (marketable) wealth lishing Ltd., 1992. between 1983 and 1989 accrued to the sold, are taxed at 28% as part of the Convergence of Productivity: Cross-Na- regular federal income tax system. But top 20% of households,” while the bot- tional Studies and Historical Evidence, capital gains that are included in a de- tom 40% of households experienced Oxford University Press, 1994. ceased person’s estate are exempt. declining wealth. The disparity was made worse by the fact that a growing European countries have a much percentage of households had zero or more extensive system for taxing house- negative net worth. This phenomenon, hold wealth, incorporating a direct tax Wolff says, creates urgency about the on wealth in addition to capital gains need for new government policies. The and estate taxes. The tax rates, howev- 1993 Congressional budget bill mirrored er, are not high — a few percents at one strategy — an increase in marginal most. In most countries, owner-occu- income tax rates for the wealthy. But pied housing is taxable, as well as these rates are still lower than at the bonds, stocks and shares, and unincor- beginning of the 1980s, and far lower porated businesses. Most countries re- than in the 1960s. And they are having quire an annual assessment of the total only a modest effect on income ine- market value of personal property, even quality. though the definition of market value is usually difficult to determine. In addi- Therefore the author pleads for a tion, most have a combination of wealth taxation system similar to the death, gift, estate and capital gains tax- Swiss direct taxation approach. He es. points to arguments in favor of such a system that go beyond its function as a The Author’s Recommendations revenue-producing and equalizing tool: 1) income alone is not a sufficient The remainder of Wolff’s book pro- gauge of well-being or the ability to pay vides data on the benefits of expanding taxes, and 2) an annual wealth tax wealth taxation in the U.S. Simulations may encourage individuals to transfer suggest that a combined income-wealth their assets from less productive to taxation system may be more equitable more productive uses. Wolff asks for a than our current income tax system. wealth tax fully integrated with a per- Wolff claims that a wealth tax is sur- sonal income tax, as in Switzerland. Po- prisingly progressive, since the higher litically, he says, such a tax on wealth taxes would be born by older house- would affect only a very small propor- holds, married couples, and whites — tion of the American population while that is, those most affluent. However, making an important revenue contribu- he pursues answers to several ques- tion.

16 Evaluation Forum n Issue 12 n Summer 1997 Features

nn actually slow economic growth. Mean- policies must address the occupational Balancing Growth while, they claim that Reaganomics training issue. contributed considerably to the in- Based on empirical evidence and the with Economic crease in inequality. authors’ concerns, a comprehensive Equality program is recommended — a mix of The Trade-Off Between Efficiency policies for encouraging long-run and Equity growth that does not sacrifice equity. For another perspective on inequali- A major theme of the book echoes Their agenda flows from their own re- ty, we briefly review the main ideas Arthur Okun’s classic, Equality and Ef- search and from a series of studies on and recommendations in Growth With ficiency: The Big Tradeoff, published in the slowdown in productivity in the Equity: Economic Policymaking for the 1975 by Brookings. Baily, Burtless and U.S. and the generally poor perfor- Next Century by Martin N. Baily, Gary Litan want us to understand that some mance of the U.S. economy. This agen- Burtless and Robert E. Litan, published policies that hurry long-run economic da responds to two major objectives: by The Brookings Institution in 1993. growth are not consistent with the goal 1. To increase economic growth as the The research supporting the book was of greater equity, and some policies sponsored by The Brookings Institu- main support for American living that improve equity may not be consis- standards. tion’s Center on Economic Progress and tent with successful economic competi- Employment, which has published eight tion in the new global era. What the authors mean is “a more books and a series of Brookings Papers rapid rate of advance in total on Economic Activity since its estab- The authors use freer trade as an ex- output...and a faster rate of increase in lishment in 1988. Growth With Equity ample of this dilemma. More open output, holding constant the amount of is the final volume published by the trade is likely to motivate companies to inputs used to produce that output.” Center. raise their productivity to remain com- This can be achieved, they believe, by petitive, but it may also reduce Ameri- While the previous articles in this is- raising labor productivity. The economy can workers’ earnings and employment could produce more by working longer sue of Evaluation Forum concentrated opportunities. Trade protection can re- our attention on economic and social hours and placing additional people in duce the damage to workers, but may poorly-paid jobs, but the added income inequality, these economists want us to jeopardize economic growth. look at the critical relationship between would be worth less than the free time slow growth in the economy and in- Despite the authors’ appreciation of given up. So it is wiser, the authors creasing inequality. What concerns Okun’s major tenet, they also believe propose, for the labor force to “work them is the long-term performance of the that growth and equity can be smarter, produce more goods, and de- American economy. achieved simultaneously. They refer to liver additional services, with the same abundant evidence that progress to- level of work effort.” When the authors The authors report that in the twen- ward both goals can be accomplished talk about economic growth, they ty-five years following 1948, output per at the same time. In fact, they see poli- therefore mean the growth in potential person grew faster than 2.5% a year. cies supporting both productivity and output, “the output that can be at- This was high enough to double the av- equity as being highly interrelated. In tained when labor and capital are fully erage American’s living standard in less the 1960s, for instance, the U.S. experi- employed and inflation is not accelerat- than thirty years. Since 1973, however, enced rapid growth and a sizeable re- ing.” Consequently, a primary goal output has grown only 1.2% a year, duction in income inequality. Policies must be to improve the efficiency with which means that the average Ameri- that increase the skill level of workers, which the nation’s labor and capital can must wait twice as long to witness reduce social problems, ameliorate con- stock are utilized. This can be accom- a doubling of his or her income. At the ditions that lead to low earned income plished, according to the authors’ agen- same time incomes have grown less among the least skilled, all contribute, da, through advances in technology, im- equal. they say, to economic growth. And provements in managerial skills, and the The American economic pie has they are convinced that policies that upskilling of workers. been getting larger far more slowly than free-up trade relations, and policies 2. To reduce income inequality, par- it did during most of the postwar era. that protect and improve the job pros- ticularly wage earnings. And the pie has been sliced more un- pects of workers harmed by freer inter- evenly. In the next decade and be- national trade, can be pursued jointly. The goal is to achieve greater equali- yond, the nation faces no more urgent ty through a reallocation of the tax bur- den or through heavier reliance on gov- an economic challenge than reversing Balancing the Equation these two trends. ernment transfers such as public The authors acknowledge that pro- assistance. The authors advocate deficit A variety of problem-solving strate- ductivity growth and improvements in reduction, focusing both tax increases gies have been suggested. Some ex- living standards in an internationalized and spending cuts on the group that perts advocate for a Japanese-style in- economy cannot occur without some has benefited most from long-term eco- dustrial policy emphasizing investment level of worker displacement. Macroeco- nomic growth, namely high-income in specific industries and lines of pro- nomic policies can minimize unemploy- Americans. The authors place more im- duction — through subsidies and trade ment and stabilize inflation, but only if portance on increased equality in pre- protection. Others suggest new invest- workers have the necessary skills to tax incomes than on a redistribution of ments in public infrastructure. Howev- give them the flexibility required to money via the tax code. In this sense, er, the authors imply that their own move to new sectors of employment. their strategy of choice is an increase in analysis shows that these tactics may Therefore they insist that public/private the job skills of low-wage workers.

Evaluation Forum n Issue 12 n Summer 1997 17 Features

nn Recommendations n It is imperative that there be in- creased investment in worker train- Economic Inequality: The authors’ comprehensive strategy ing, particularly for those lacking the involves a large number of recommen- kinds of labor market skills that are A Comparative View dations under several policy areas. Un- associated with higher-wage employ- derlying themes are: ment. In 1995, the Organization for Eco- n The most effective way to increase n Free trade and its openness of for- economic growth is to increase the nomic Cooperation and Development eign goods, technology and capitol (OECD) in Paris published a landmark investment portion of national out- must be emphasized. put — that is, increasing private/ report, Income Distribution in OECD Countries: Evidence from the Luxem- public spending that is oriented to n Job losers must be afforded assis- future needs. tance in moving from shrinking to bourg Income Study. The report pro- growing sectors of the economy to vides dramatic information on postin- n The best strategy for equalizing in- reduce the economic costs of dislo- dustrial societies in which western comes is to increase individuals’ and cation. youth are growing to maturity. The families’ net savings, sufficiently book-length report details the research that these savings rise faster than Some of the recommendations regarding questions, methodology and results as- private/public investments. labor policies are presented in Figure 1. sociated with a series of analyses of data collected by the Luxembourg In- come Study (LIS), supplemented by in- formation from studies carried out by Figure 1 n Edward N. Wolff’s Labor Policy Recommendations member countries of the OECD.

General Recommendations Specific Recommendations The LIS was begun in 1983 under The improvement of skill training within Encouragement to schools and employ- the sponsorship of the government of schools. ers to provide high school transcripts to Luxembourg and the Centre for Popula- potential employers. tion, Poverty and Policy Studies in Wal- ferdange. It is being funded on a con- The requirement that all students take na- tinuing basis by the center and the tional standardized tests. International Network for Studies in The tying of salaries and promotions for Technology, Environment, Alternatives, teachers to the performance of students and Development (INSTEAD), and by on standard tests. the National Science Foundations in The improvement of the skills of the cur- The establishment of a public/private enti- member countries of the OECD. The rent labor force. ty to develop tests and credentialing sys- main objective has been to create a da- tems for noncollege occupations. tabase containing social and economic The institution of a pay-or-play training re- data gathered through household sur- quirement for non-college-education veys in 25 countries, which can serve workers. as a source of information for compara- tive studies. The development of improved R&D pro- The development of a more systematic grams for public training efforts. evaluation effort. The OECD report is authored by The encouragement of firms to link work- The creation of a system for measuring three economists who are associates of er pay with company-level performance. performance. the INSTEAD — Anthony B. Atkinson of Oxford University in the United King- The upgrading of employment services. The infusion of employment services with dom, Lee Rainwater of Harvard Univer- updated occupational information. sity, and Timothy M. Smeeding of Syra- The establishment of a non-cause-related The conversion of the UI system to a cuse University — who directed the program of earnings insurance. multi-use system for different kinds of un- analyses. The studies were conducted employment. in cooperation with OECD’s Economics and Statistics Department and its Edu- cation, Labour and Social Affairs Direc- torate. The purpose of the analyses was to examine income distribution across OECD countries and compare the ex- tent to which income is dispersed be- tween households, adjusting for the size of households, international differ- ences in real income, and (where suffi- cient data were available) the influence of property and employment incomes while accounting for direct taxes and cash transfers.

18 Evaluation Forum n Issue 12 n Summer 1997 Features

The authors are quick to point out n The large-scale demographic chang- clusion is that the Scandinavian coun- that not all the data available in the LIS es occurring, such as the growth of tries, Benelux and Germany exhibited databank cover the same time periods single parenthood. the least relative inequality. The highest or are equally extensive. For eight major levels of relative inequality were record- n The changing form of taxation and OECD countries, however, a substantial ed in the U.S., Switzerland and Ireland. income transfer policy, such as the analysis of the level and trends in ine- There was a rise in the 1980s for the lowering of the top income tax quality was possible, and the results majority of countries studied, but this brackets in several countries and the could be compared to national studies was not universal. Increases in mea- reduction in benefit coverage and conducted within each of these coun- sured income inequality were largest in levels. tries. The researchers also acknowledge the Netherlands, Sweden, and particu- that there were a number of expected n The world-wide increase in returns larly in the United Kingdom and the barriers to data comparability. Never- on capital income and changes in United States. theless, the LIS is viewed consistently the distribution of asset ownership, The first comparative approach re- by experts as the most comprehensive including privatization programs. ported differences in median equivalent international study ever conducted in n Immigration, technological change, incomes per adult. For example, in the the industrialized nations. Furthermore, and the transformation of Eastern U.S. the median equivalent income per it avoids some of the definitional prob- Europe. adult was $13,364 in 1986. For a family lems in previous comparative analyses, of four, this corresponded to a total in- and is highly relevant given the grow- Although the authors emphasize come of $26,728. The equivalent in- ing interdependence of economic, so- these changes, and independent stud- come per adult at the lower quartile, cial and political trends in the countries ies have suggested that all of these 25% up from the bottom of the income involved in the study, and the compel- forces impinge on the distribution of in- distribution, was $8,240; the 25th per- ling interest in cross-national research come, influencing the nature and level centile, expressed as a percentage of on income distribution. of income inequality, they do not credit the median, was 61.7%. The decile ra- their analyses with estimating cause-ef- tio in the U.S. was the largest value re- fect relationships. Accounting for the Trends Leading to Increased corded in the income distribution for distributional effects of these trends, Policy Interest in Economic OECD countries. The bottom decile was they say, is simply not currently possi- only slightly over a third of the median, Inequality ble. It is now critical to explore similari- compared with around 45% in Australia ties and differences in industrial na- The authors are straightforward also and Canada, and values in excess of tions’ resource distribution patterns about barriers to data comparability: 55% in Belgium, Finland, France, Ger- many, Luxembourg, Norway and Swe- over time, and to understand how the n All comparisons were made in terms den. However, differences across coun- market and demographic forces and of intra-country relative measures, tries were less dramatic at the top public policies affect the relative eco- and did not have as a goal to seek percentile. nomic status of various groups within absolute comparisons of levels of these societies. well-being in different countries. Another approach was to express differences in terms of what proportion Such issues directly influence the fu- n Certain microdata sets created prob- of the population is below specified ture opportunities and potential long- lems regarding “interior vs. exterior” percentages of the median — for exam- term life quality of youth in these coun- comparability. tries. Some of these influences had ple, how many people are living in their origins outside government; others n Using information covering only two households with disposable incomes appear correlated with government pol- years in some cases made it difficult below half the median. If 50% of aver- icies. The authors cite the following im- to identify general trends. age income is used as a standard defin- ing “low income,” and if “average” is portant factors operating across OECD In summarizing the section on data measured by the median, then the U.S. countries: quality, the authors emphasize the fol- stood out with 18.4% of the population lowing: 1) full comparability is impossi- n The different experiences of these living in households below 50% of the ble, 2) adopting a common set of defi- countries regarding macroeconomic national median. The proportion with nitions means that estimates for any performance, particularly in the “modest” incomes, between 50% and one country may be less satisfactory 1980s. 70% of the median, was lowest in the than if each were isolated, and 3) stan- U.S. and largest in the U.K. n The expanding inequality in the dis- dardizing through a common approach tribution of earnings in several OECD does not necessarily achieve compara- At the same time, in the U.S. the countries. bility. Nevertheless, it is clear that the proportion in “well-to-do” households researchers cautiously identified the (above 150% of the median) was more n The general decline in the participa- tion of men in the labor force, and major problems and sought to adjust for than double that in Sweden. In the the increase in early retirement, at them to the extent possible. U.S., U.K., Italy, Ireland and Australia, the same time female labor force about a quarter of the population is participation has been increasing The Main Findings from the within the middle of the income distri- and female wages have risen relative Analyses bution; in Sweden nearly 40% is in to male wages. that range. In the U.S. there are more Several approaches are used for re- households in both the lower and up- n The aging of the population, creating porting comparisons across seventeen per income categories; in Sweden, few- strong budgetary pressures on na- postindustrial nations. The general con- er in those categories. tions’ safety nets.

Evaluation Forum n Issue 12 n Summer 1997 19 Features

A third approach was to report dif- size, and 4) the kind of adjustment counted only direct taxes such as na- ferences in terms of shares of total in- used to represent inequality. tional income taxes and payroll taxes come. Using this measure, the U.S. paid by employers and the self-em- The researchers recognized that the again reveals considerable income ine- ployed. strategies used were not a perfect solu- quality vis-a-vis other western societies. tion to the definitional problem. Never- Average income in the U.S. would have theless, they were resolutions support- Some Conclusions to be higher by 50% if there were a ed by the 1977 United Nation’s more equal distribution of income. In Despite understandable data quality Provisional Guidelines. This directive looking at trends in the distribution of and comparability problems, the OECD outlined a standard structure for defin- income over time, the majority of OECD analyses take a giant step toward a bet- ing and collecting income distribution countries showed a rise in inequality, ter understanding of the differences in statistics via national reporting systems. most particularly the U.S. and the U.K. income inequality across postindustrial For example, the authors acknowledged societies. The comparisons make a that the line between market and gross statement about the relationship be- The Unique Methodology Used income was sometimes artificial. The tween social democratic principles and Cross-national comparisons are fre- only serious concern remaining was the economic equity. This statement is quently misleading because the data do exclusion of employers’ contributions alarming with respect to the inequali- not reflect comparable definitions of from wages and salaries to government ties evidenced in the U.S. in the late the measures used to study key vari- occupational transfer programs, and 1980s, given the social polemics about ables. In the analyses of LIS and the rental income from owner-occupied egalitarianism. It is also an important supplementary national data, measure- housing. Also, measures of some vari- commentary on our form of capitalism ment issues were given unusual atten- ables were not available or complete. and the role of the central government. tion. The major issues were 1) the defi- For instance, infrastructure, defense And it provides a context for discussing nition of income, 2) the definition of and protective services were excluded, problems facing American youth, which the unit and population studied, 3) the as well as health, education and other should be taken very seriously. kind of adjustment made for household in-kind subsidies. And the analyses

From a 1990s report prepared by the Office of Technology Assessment, U.S. Congress In 1993, approximately 7.3% of the among eighth graders was slightly ficient work in the eighth grade. In U.S. population was young adoles- higher in 1992 than it was in the 1990, 7% of the eighth grade class cents, ages ten through fourteen. Of 1970s. However, these performanc- of 1988 (most of whom were then fif- these, approximately 20% were liv- es have not improved enough to teen and sixteen) were dropouts. By ing below the federal poverty line. keep pace with the higher level of their senior year, 12% of this class By the year 2000, more than one- skills required in a global economy. were dropouts. Dropout rates vary third of all young adolescents will In 1992, only 28% of eighth graders by students’ race/ethnicity: white be members of racial or ethnic mi- attained the proficient level in read- (9.4%); black (14.5%); Hispanic norities. To compete in the global ing, representing solid academic (18.3%); Asian/Pacific Islanders economy of the twenty-first century, performance at grade level. Two per- (7%); and American Indian (15.4%). America will need all of its young cent read at or above the advanced The personal economic conse- people to be healthy and well edu- level, representing superior perfor- quences are profound. High school cated. mance. The remaining 70% demon- dropouts are unlikely to earn suffi- The average proficiency in sci- strated partial mastery of the knowl- cient wages to support themselves ence, mathematics, and writing edge and skills required for pro- or their families.

20 Evaluation Forum n Issue 12 n Summer 1997 Commentary Commentary

Introduction His major areas of concentration Brandeis University. He is also the se- have included labor market issues, in- nior faculty member associated with In this issue we place a magnifying come distribution, and public tax and the Center for Human Resources, the glass over youth issues. We have transfer policies. His recent research director of the Heller Program on Inno- asked the following group of experts has focused on the sources of increas- vations in Social Policy, and the direc- on youth development, programming ing wage and income inequality in the tor of the Rothman and Giddon Fami- and evaluation to respond to a set of U.S., the influence of international lies Project on Media, Health and questions about youth problems and trade on income inequality, the job Social Policy. Most of his research and possibilities. market prospects of public assistance policy work has been in the areas of recipients, and the reform of social in- education, training and employment for David E. Brown is a senior policy surance in Central and Eastern Europe disadvantaged youth, dropout preven- analyst with the Center for Policy Re- and developing countries. During his tion, and welfare reform. He was co-di- search at the National Governors’ As- distinguished tenure at Brookings, he rector of Career Beginnings, a national sociation. Long interested in youth is- has co-authored and edited several mentoring program for disadvantaged sues, beginning with a high school books and authored numerous articles youth, and is involved currently with an- internship with one of New York’s Youth on critical social policy issues. other large-scale youth project, Summer Bureaus, he majored in Political Sci- Beginnings. ence and Urban Affairs at American Susan P. Curnan has been director University and received a graduate de- since 1990 of the Center for Human Hahn has evaluated a range of gree in public administration from Resources in the Heller Graduate youth initiatives for the Rockefeller, Baruch College as a National Urban School for Advanced Studies in Social Ford, Clark and Lilly Foundations, re- Fellow. Policy at Brandeis University. She is cently completing an evaluation of the also an associate professor and chair Quantum Opportunities program which In the early 1980s, Brown adminis- of the Heller School’s graduate pro- focuses on youth development. He has tered federally-funded youth employ- gram in Management of Children, been principal investigator for a num- ment programs in New York and later Youth and Family Services. Her major ber of multi-year national youth pro- coordinated education and work expe- work is in translating research and ex- grams, and is currently principal evalu- rience projects targeting economically pert opinion into “best practices,” the ator for a national business/education and educationally disadvantaged out- design and management of innovative partnership program, a national net- of-school youth in New York City. In the national and local evaluations, and work of education and entrepreneur- late 1980s, he developed an employer- consultation on issues such as educa- ship programs, a high school program based career exploration and job tional reform, workforce preparation in Baltimore, and a project to examine preparedness program for at-risk teen- and community development. Her inter- federal youth policies. agers. Working for the Maryland De- ests extend also to strategic planning partment of Juvenile Services, he for- Robert J. Ivry is Senior Vice Presi- regarding youth issues and the moni- mulated policies and legislation to dent for External Affairs and Develop- toring of a clearinghouse and national improve the responsiveness of the ment at the Manpower Demonstration electronic network on children, youth state’s juvenile justice system. At the Research Corporation (MDRC). He is and families. Center for Policy Research, Brown is an expert on a range of policy issues, concentrating on social policy, legisla- Curnan has been principal investi- particularly youth employment policy. tion and new initiatives in youth develop- gator of a four-year youth research At MDRC he is responsible for devel- ment, school-to-work, job training, youth project funded by the U.S. Department oping and implementing many of the violence, and community service. of Labor which synthesized research research firm’s major demonstration findings from a number of social sci- projects, and for disseminating the re- Gary Burtless is a Senior Fellow in ence disciplines. She was principal sults and recommendations flowing the Economic Studies program at the consultant in the conceptualization and from evaluations of these demonstra- Brookings Institution in Washington, implementation of the Department of tions to policymakers and practitioners. D.C. A Yale and MIT-trained economist, Labor’s Youth Fair Chance initiative, He has been involved with a number of Burtless held positions in both the U.S. and was chief developer of the Sum- youth demonstrations such as JOB Department of Labor and the U.S. De- mer Beginnings National Work and START, New Chance, Career Begin- partment of Health, Education and Wel- Learning Network. In 1982 she was nings and Project Transition — their fare in the late 1970s and early 1980s elected an Outstanding Young Woman development, implementation and — in the office of the Assistant Secre- in America in recognition of her profes- evaluation. He was instrumental in tary for Policy, Evaluation and Re- sional and community service accom- broadening MDRC’s research focus to search in DOL, and the Office of the plishments. include the evaluation of innovative Assistant Secretary for Planning and school-based reforms beginning with Evaluation in HEW. He has been at Andrew B. Hahn is Human Services the School-to-Work Transition Study. Brookings since 1981, with the excep- Research Professor and Associate And he led the team conducting an ex- tion of 1993 when he served as Visiting Dean of the Heller Graduate School for perimental evaluation of Career Acad- Professor of Public Affairs at the Uni- Advanced Studies in Social Welfare at emies, the first such study of one of the versity of Maryland.

Evaluation Forum n Issue 12 n Summer 1997 21 Commentary

options under the school-to-work legis- merly founder and partner of Grinker, lation. Walker and Associates, Inc. in New Expert Panel Ivry’s career has combined project York, a private sector research firm. Be- Responses management with policy research. In tween 1981 and 1983 he was Senior the 1970s he managed the mayor of Vice President for program develop- Baltimore’s Youth Services. Moving to ment at the Manpower Demonstration Research Corporation in New York, Question: What major problems do MDRC in 1980, he took part in the you feel American society is facing in and held other managerial positions in management of the 1980s’ Youth Incen- the 1990s regarding youth development, MDRC between 1974 and 1983. tive Pilot Projects, the largest national education, occupational training and youth employment demonstration ever A graduate of Yale Law School, employment? funded by the U.S. Department of La- Walker began his career with a New bor. He has since been involved with a York law firm, then became president of David Brown range of policy initiatives in education a legal service firm, and subsequently and employment for disadvantaged served in supervisory positions in the Look at this quote: youth, school dropouts, teen parents Vera Institute of Justice in New York. He and youth in welfare families. is the author of a number of policy What do our children and youth analyses and evaluations focusing on face today? Frustration, insecuri- Christopher T. King is Research ty, fear, despair, resentment, con- Scientist and Associate Director of the youth, and a consultant to The Ford Foundation and other national founda- fusion, and lack of opportunity. Center for the Study of Human Re- We must not let these difficult sources in the Lyndon B. Johnson tions. He currently serves on The As- pen Institute Roundtable on Compre- years rob our young people of School of Public Affairs, the University hope. of Texas at Austin. At the Center since hensive Initiatives for Children and 1980, King has conducted numerous Families, and is a member of the Urban This statement by the General Secre- research projects in the areas of edu- League’s Advisory Committee on Youth tary of the National Child Labor Com- cation, training, welfare and employ- Development. mittee describes contemporary circum- ment. His current research focuses on Alan Zuckerman is Executive Direc- stances, yet it was made on April 26, the implementation and impacts of the tor of the National Youth Employment 1939 during the Conference on Children JOBS program and related welfare ini- Coalition, which involves eighty mem- in a Democracy. This national meet- tiatives, and on outcome measurement ber organizations. The Coalition’s pur- ing — hosted by the first U.S. Secretary and school-to-work issues. poses are to improve training capacity of Labor, Frances Perkins — was and prepare youth to become produc- opened by President Franklin D. He has taught labor economics, hu- Roosevelt and First Lady Eleanor man resource management and policy tive workers, to serve as a clearing- house for information about successful Roosevelt. The ominous rise of fascism analysis at the University of Texas, and communism across the globe in the Southwest Texas State University and youth development programs, to pro- vide a forum for exploring alternative 1930s and the combination of wide- the University of Utah. He was Chief of spread youth idleness, unemployment Research, Demonstration and Evalua- youth employment policies and strate- gies, and to create a support network and despair in America heightened tion for the Texas Department of Com- anxieties about the commitment of munity Affairs and the Governor’s for the youth development organiza- tions. American youth to democratic princi- Office, initiating an experimental evalu- ples. However, many of the pressing ation of Texas’ dislocated workers pro- Zuckerman has developed policies youth problems addressed by the 1939 gram. In addition he was principal edi- and implemented programs in employ- conference were ameliorated in large tor for the U.S. Department of Labor’s ment and training for the U.S. Depart- part by the onset of World War II. Most Advisory Committee report, Working ment of Labor, the Council of the young people were, for better or worse, Capital, and authored Cross-Cutting Southern Mountains, the Opportunities actively engaged in the war effort. Sub- Performance Management Issues in Industrialization Centers of America, sequently, many of the young of that Human Resource Programs for the Na- and the Home Builders Institute — and era who survived the war were the tional Commission for Employment Pol- Temple University, where he has beneficiaries of the greatest single fed- icy. He has served as a consultant on taught courses in manpower and youth eral investment in postsecondary edu- performance measurement and man- development. He has been a consult- cation — the GI Bill — and the unprec- agement issues for a number of gov- ant to the governors of New Jersey and edented postwar economic expansion. ernment and private sector agencies Pennsylvania, the President’s Commit- and organizations. tee on Manpower, the National Com- Today, many of our young people are once again facing frustration, insecuri- Gary Walker served as president of mission for Employment Policy, com- ty, fear, despair and resentment. With- Public/Private Ventures in 1995, and as munity development agencies and out promising prospects for their fu- its executive Vice President between public housing authorities. A long-time tures, hope has again become an 1986 and 1994. Public/Private Ven- advocate for improved efforts to assist increasingly rare commodity among tures is a private nonprofit research young people in developing their many young adults, particularly those and demonstration organization in potential, Zuckerman has written nu- residing in urban and rural communi- Philadelphia specializing in youth pro- merous articles on youth employment, ties. Increasing levels of youth crime gramming and evaluation. He was for- service delivery and computer commu- nication. and violence, and the seemingly inex- plicable allure of gangs and paramilitary

22 Evaluation Forum n Issue 12 n Summer 1997 Commentary

hate groups among many youth, are There can be little doubt that this cies to block grant federal welfare and indicative of a prevalent loss of hope. school-to-work “system” leaves many workforce development dollars to states States have experienced a continued youngsters unprepared to earn a good and localities is taking the country into rise in the social and fiscal costs of the living. uncharted waters and runs the risk of nation’s inability to effectively absorb exacerbating these problems. The unin- many of its youth into the economic Andrew Hahn tended consequences for the health and cultural mainstream. Drug-related and economic well-being of young peo- murders, gang activity and malicious The major problem we face regarding ple could be devastating. Youth pro- hate crime have plagued communities. the “human resource” challenge is that grams are likely to get caught in the tri- High rates of teenage pregnancy con- as a nation we have not and do not ple squeeze of reduced funding, tinue to arrest the development of take seriously the development of a lo- non-targeted funding for youth from many adolescent girls, foster irresponsi- cal infrastructure to help young people low income families, and the competi- bility among teen fathers, and impair make developmentally appropriate tran- tion for resources between programs for the development of the next generation sitions into the mainstream of society. adults and programs for youth. Since of children. The job training/youth development/ young people are not a voting constitu- second chance education “field” is real- ency, they are likely to get short- This time, however, there is no na- ly a constellation of small professional changed on resources and services. tional challenge as formidable as World efforts operating on the margin of most War II or a federal program as ambi- communities’ efforts. While this gener- One of the challenges youth advo- tious as the GI Bill on the horizon capa- alization certainly has many exceptions, cates face is to change the public ble of once again positively engaging the fact is that we do not take skills mindset, to view young people as a re- vast numbers of the nation’s young training in the public and not-for-profit source and asset to be nurtured and people. Furthermore, America is no spheres very seriously. It is an under- cultivated rather than a liability to be longer the singular economic power- capitalized field serving only a fraction ignored or vilified. What often gets ob- house it was after the war. Today, U.S. of the universe of need and eligibles. scured in the blizzard of negative youth companies must compete aggressively There are few professional certification statistics is the incredible resiliency of with their foreign counterparts to retain standards, and program quality mea- young people, despite overwhelming their share of international trade. sures are rarely widely used and ac- circumstances. For example, contrary to cepted. Evaluations of programs, with the stereotype of teen parents, there is Gary Burtless some important exceptions, continue to evidence that many young mothers on reinforce the image that these programs welfare remain in school, work and en- Young people in the United States face are ineffective and do not matter. roll in training programs despite facing one main problem in getting estab- formidable personal and situational lished in the work force — finding a Skill training for the disadvantaged problems. Public support for education source of training that is widely recog- has always been a small enterprise. It and crime prevention provide a rallying nized as effective. An increasing per- is now in retreat, and even among point for policymakers to overcome centage of young people attempt to some advocates there is the feeling that their shortsightedness and enact poli- solve this problem by enrolling in high- it might be better to experiment with cies which support the development of ly-regarded two and four-year colleges. consumer choice models (e.g., skill young people from adolescence to This solution probably works for the grants) than invest in “programs.” Pub- adulthood and from school to work. majority of students who enroll and lic education enjoys a property tax form complete college. But for many teenag- of financing; in job training the finan- There are several compelling reasons ers college is not a satisfactory solution cial base and very mission of the enter- why investment strategies are impera- to their problem. They may have failed prise is still debated. This shows how tive. First, there is a broad-based con- to perform well in secondary school much work is still required to get work- sensus that our nation’s productivity classrooms, so it is doubtful whether a force development policy and programs and international competitiveness hinge college classroom will offer a better firmly embedded in our local communi- on helping young people — our future venue for training. What these young- ties. labor force — become skilled workers. sters need is a source of hands-on oc- This is one of the reasons why the cupational training that is highly re- Rob Ivry School-To-Work Opportunities Act of garded by employers. There are very 1994 passed with bipartisan support, few sources of such training outside As we approach the next millenni- and why the nation’s governors agreed the college classroom. um, the state of America’s youth is pre- at the education summit meeting in carious. Stories abound in the press March 1996 to encourage states to set Youngsters who do not fare well in about youth alienation and apathy, ju- skill standards. Second, the public costs college do not have good training op- venile crime, stagnant or deteriorating associated with disconnected youth, in portunities in the private workplace. test scores, persistently high dropout terms of outlays, lost productivity and Few employers are willing to take the rates, youth joblessness, and teenage reduced taxes are exorbitant. Third, risk of investing in high school dropouts pregnancy. The largely discouraging re- the public investment in knowledge de- or high school graduates who are in sults from studies of youth employment velopment over the past twenty years their late teens or early twenties. As a and training programs have overshad- has the potential to pay dividends if result, many job market entrants who owed the success stories and are often the terms of the debate can be lack a college degree are destined to go generalized to indict all social invest- changed from “why have programs from job to job without obtaining much ments for young people. Proposed poli- failed?” to “how can youth programs occupational training in any of them.

Evaluation Forum n Issue 12 n Summer 1997 23 Commentary

be more successful?” While the results these jobs, nor are they maintaining the mies have so clearly become inter- of most youth programs have been dis- institutional memory of them which meshed at all levels and in all dimen- couraging, especially for young people older workers brought to bear, so that sions, and people’s control over their who have dropped out of school, there young workers have access to the type lives diminished. Note that some of the are some beacons of proven effective- and level of human capital investment fastest growing groups in our country, ness. The challenge is to promote poli- tools within firms which they would now and for the foreseeable future, are cies which enable states and localities have had before. the very ones who are the focus of this to build upon and expand these suc- intolerance. Second, and related, is that even cesses while encouraging further exper- most workers who have been holding Educational institutions are now ex- imentation, so that the repertoire of onto their jobs are watching as their pected to play roles which would have proven programs can grow. earnings decline in value. A now well been inconceivable just a generation or documented and expanding gap exists two ago. Schools must educate for de- Chris King between those in the top reaches of mocracy and citizenship as well as Undoubtedly, every generation of the labor market and virtually everyone playing a major role in preparing youth youth has had to face its own unique else (Burtless, Danziger, Gottschalk, for work and careers. They also are set of major problems since the begin- and many others). Youth must be fac- serving as health and parenting skills ning of time. In this century, they have ing increasing pressure in poor families educators, child care facilities and so- had to deal with massive urban migra- to drop out of school to help make ends cial halls. tions and the disappearance of rural meet, especially in light of the dimin- family life, an all-encompassing eco- ishing and less dependable safety net. Gary Walker nomic depression, world wars and divi- Third, whether at the national, state For a large and increasing number of sive “conflicts.” This generation is now or local level, government budgets are youth, the basic forces in their “natu- encountering its particular set of prob- severely squeezed. Not surprisingly, ral” surroundings — family, neighbor- lems. Coming as they do after a governments are attempting to divest hood, friends, school, employers, con- uniquely prosperous era in our history, themselves of, or cut back on, some of nections to the larger world — are these problems must appear especially their primary functions. The enormous weakening simultaneously. Quite unwit- daunting. and unprecedented peacetime deficits tingly, our society has become steadily First among these problems is the of the 1980s are at the heart of this less hospitable to the healthy develop- fact that a job no longer carries with it phenomenon. The recent school-to- ment and transition to adulthood of its what it did for these young people’s work initiatives are a prime example of youth. That is the most fundamental parents and their parents’ parents. For this trend. The federal government es- problem we face. most, a job once meant a career, a sentially has promised only “seed” In addition, our society has in its sense of identity and belonging, a money for this initiative, explicitly ex- historical and cultural roots a deep am- steady rise in earnings — and an array pecting the financially-strapped states bivalence about direct government in- of associated health, vacation and other and localities to step in with full subse- tervention in social issues. That ambiv- benefits, as well as long-term economic quent funding. This is despite wide- alence is particularly evident regarding security at work’s end. Getting a good spread recognition that poor and un- social and economic equity policies. We education, working hard and playing structured transitions from public do not consistently exercise the com- by the rules brought these things and school to higher education or to work mitment or good sense necessary for more. Today, young people are watch- on the part of our youth is truly a na- government policies and resources to ing as most of this picture fades from tional problem of considerable magni- help rebuild the basic positive forces in the screen within a time so short they tude (Commission on the Skills of the many youths’ lives. We have a hard can catch it even with the shortest of American Workforce’s 1990 report, time seeing public social policy as an attention spans. America’s Choice; and Ray Marshall and Bob Glover, 1994). integral part of what it takes to keep a Peter Capelli (“Rethinking Employ- complex free market and individual-ori- ment,” British Journal of Industrial Re- Fourth, the nation seems to be mov- ented society hospitable to healthy lations, forthcoming) and others have ing into a period of intolerance and youth development. Worsening condi- suggested that career paths by occupa- mean-spiritedness. This is unprecedent- tions for youth, and an ambivalent atti- tion have become truncated, and ed in times of high growth in Gross Do- tude about using public policies and where they still exist their beginning mestic Product (GDP) and productivity, funds to improve those conditions, are rungs are no longer inside the firm. and relatively low levels of unemploy- a serious pair of obstacles. Rather they are located within tempo- ment and inflation. Any number of rary services. Many young (and not so groups appear to be singling out minori- Alan Zuckerman young) workers now find that they ties, homosexuals, immigrants, welfare must set out on their careers working recipients and single mothers and their Andrew Sum, of Northeastern Uni- as contingent workers hoping to secure families as objects of scorn, derision versity, observes that economic adoles- a more steady second-rung job with a and out-and-out hatred. Ironically, cence has been extended until the age “real” employer. Moreover, as they go national and state executives and legis- of thirty. This is happening at a time on, employers are no longer providing latures are rapidly rushing to hold indi- when wages for all but the top 20% of the training which once accompanied viduals and families “personally respon- wage earners are going down and sible” for their social and economic young people are working fewer hours. misfortunes, even as the world’s econo- As a result, conventional wisdom about

24 Evaluation Forum n Issue 12 n Summer 1997 Commentary

young people’s entrance into the labor stroys motivation and limits aspirations. imperatives of adolescence is the need force must be tempered by the cruel re- Without faith in the future, many to be valued, to have a useful, mean- ality of the contemporary labor market. young people drop out of school and ingful and respected role. Since in According to Dr. Sum: fail to invest in building their work-re- many instances the family and the lated skills. community do not provide such oppor- n There are 3 million dropouts ages 16- tunities, young people seek and find 24 — 2 million dropouts are unem- validation, status and respect from their ployed. Question: What socio-economic, polit- peer group. However, when unguided n The average number of weeks ical, demographic and cultural trends by parents or other adults, the norms, worked by 17-21 year olds has in American society do you think have values, expectations and behaviors of dropped slightly for all young people contributed most to the development or the peer group often undermine years and plummeted for African Ameri- exacerbation of these youth problems? of parental socialization. For most can male dropouts to 20.9 weeks a youth, it is within the peer group that year. (The average for African Ameri- David Brown relationships with the opposite and can male high school graduates is same sex are developed, new roles are 35.3 weeks compared to 40.5 weeks Over the past few decades, the na- tested, status is achieved, indepen- for all high school graduates.) tion has responded to the problems of dence asserted, and identity eventually its most vulnerable youth by initiating established. Concurrently, much of the n Real annual earnings for 17-21 year myriad crisis-driven and narrowly-fo- antisocial and self-destructive behavior olds fell between 1973 and 1994 by cused prevention and intervention pro- engaged in by adolescents occurs with- 38% (to $4,968) for male dropouts grams. A recent General Accounting Of- in the context of peer group activities and 25% for all male high school fice report asserts that “The federal or before a peer audience. These be- graduates (to $10,180). response to the problems of delinquent haviors, though often considered devi- and at risk youth is currently manifest- n For 20-29 year olds, mean earnings ant by adult standards, generally fall for male dropouts fell 42% (from just ed in 131 separate programs adminis- within the accepted boundaries of the under $20,000 in 1973 to $11,223 in tered by 16 departments and agencies. peer group’s norms. The need to be ac- 1994). Many of the programs are authorized to cepted and to achieve status among fund multiple services and are targeted peers has proven to be a powerful and, n According to Doug Besharov of the towards multiple youth groups, creating to date, virtually unchallenged force American Enterprise Institute, 10% the potential for program overlap and during adolescence. of America’s youth (ages 16-23) are duplication of services.” “disconnected” — not in school, not The continued weakening of family in the military, not working, not mar- Most of these programs target de- and community social support systems, ried for at least one year. fined segments of the nation’s young coupled with the dearth of programs or people: those exhibiting behaviors or strategies promoting overall youth de- n For women, the rate is 34% of Afri- experiencing problems commonly asso- velopment, have placed many of Ameri- can Americans, 21% of Hispanics, ciated with a specific adolescent “pa- ca’s young people “at risk.” Due partly 9% of whites. thology” (e.g., delinquency, substance to family breakdown and increased abuse and pregnancy), or those n For men, the rate is 23% of African feelings of alienation, many youth are Americans, 13% of Hispanics, 7% of deemed at risk for developing such be- engaged in self-destructive behaviors, whites. haviors or problems. Interventions fre- including delinquency, substance quently occur after a problem has sur- abuse and early childbearing. Further- n The long term impact of being a faced, rather than seeking to prevent more, increasing numbers of the na- “disconnected youth” is: the problem from occurring in the first tion’s youth are growing up in poverty – 50% are poor. place. They often fail to address the and with dysfunctional families or sin- psychosocial developmental needs of gle-parent families. Many have little – Income 50% lower than never- youth. This categorical, uncoordinated hope for a better life and limited oppor- disconnected. and fragmented maze of federal pro- tunities to develop to their full poten- – Less likely to have health insur- grams confounds local service delivery tial. As the prospects for decent-paying ance. and promotes delayed intervention rath- jobs have eroded for many poorly edu- er than holistic prevention. Moreover, cated youth, so has their faith in the – Less likely to have ever worked these programs have not significantly system. However, the increasing reli- for a full year. reduced the problem associated with ance of states on secure confinement to Lower wages and a later entry into adolescents. In many communities, the address the failures of our youth-serv- full-time employment makes it difficult problems have increased despite inten- ing systems, coupled with the cuts in to achieve the American Dream — get- sive interventions. funding for preventive programs, are ting a job, raising a family and living Since the dawn of this century, we depleting state resources, just as the happily ever after. For too many young have continued to postpone the age at nation’s youth population is about to people, work is sporadic and low pay- which young people are expected to as- surge again in the latter part of the de- ing, leading nowhere. This reality de- sume adult roles. Yet one of the most cade. commonly overlooked developmental

Evaluation Forum n Issue 12 n Summer 1997 25 Commentary

Gary Burtless complex to disaggregate genuinely Urban schools have been resegregated these relationships, although social sci- based on “white flight” and residential The problems faced by non-college- entists enjoy trying to do so. I do think housing patterns. The enrollment of bound youngsters have been exacerbat- a trend worth noting is how the pre- students of color has grown to about ed by the strong growth in demand for ceding social and economic movements 75% in urban schools. Poor urban highly qualified workers. This trend is have touched the program sphere. Con- youth are physically isolated from job evident in widening wage disparities sider that too many talented people growth opportunities in the suburbs, between workers with college degrees have outmigrated to the suburbs, afraid and public transportation is inadequate and workers who have not received any to work with young people in our inner to enable many young people to get to education beyond high school. The city neighborhoods. Program staff in these jobs. These youth often know trend is also evident in widening dis- workforce and community programs are few people who work at steady, full- parities between the wages received by often disconnected from highly es- time jobs, and thus lack access to job workers who have just entered the la- tranged youth, and do not know how networks that middle and upper class bor force and workers who have had best to help them combine hard skill youth use to connect to jobs. fifteen or more years of job experience. training with soft and life skill training. Fourth, the recently enacted welfare Both educational qualifications and The “background” problems of family reform legislation may place added lengthy work experience are more high- life and poverty are so severe that strain on poor families and place added ly prized today than they were two or many young clients bring a bundle of responsibilities on young people from more decades ago. problems that “single purpose” short- households receiving welfare. In the term programs have a difficult time ad- It is more important than ever that presence of time limits on cash assis- dressing. The point is that the same workers obtain good occupational cre- tance and work requirements, young trends that can be identified for the dentials, but young workers without a people may have to watch younger sib- country as a whole affect workforce de- college degree find it harder than ever lings, or drop out of school to work, in velopment programs. The latter are to find employers willing to invest in order to contribute to household in- both part of the problem and the solu- the kinds of credentials that actually come. Finally, the demographic trends tion. would help their young workers. Young reveal a forthcoming bulge in the youth people with less than a college degree population, with an increasing propor- have paid a heavy price for this defi- Rob Ivry tion from low income families. Unless ciency in our training system. They Several interrelated factors have con- the public commitment changes, the have suffered steep wage declines in tributed to the increased vulnerability problems confronting young people in comparison with youngsters who had of America’s youth. First is the rapidly schools and in the labor market could similar levels of education but who en- changing labor market. Over the past worsen. tered the work force twenty or more several decades, the relatively high- years ago. This has put heavy pressure paying, low-skill manufacturing jobs on parts of the American social safety Gary Walker that provided gateway employment for net that serve young adults and their The fact that it takes increasingly generations of young Americans, in- children. Over time, the nation has more time, energy and good luck for cluding those lacking a high school di- seen growing caseloads for food adults to make a decent living is the ploma, have disappeared. The skill re- stamps, AFDC, Medicaid, and other major social trend that weakens the ba- quirements for today’s entry-level jobs public benefits targeted on poor fami- sic forces which support healthy youth are more demanding and the so-called lies with children. development. Adults are not focusing, “youth jobs” which do exist, generally either privately or publicly, on what in the retail industry, are often part- their youth are facing and what they Andrew Hahn time, temporary, low-wage positions should do to assist them, because their that do not pay benefits. This accounts I am not particularly skillful about own commitments, stresses and worries for a widening income gap between identifying the trends that have con- have increased. This relatively recent young people with post-secondary de- tributed the most to the development economic squeeze on adults exacer- grees, young people with high school of social problems and the exacerbation bates a longer cultural trend toward diplomas, and high school dropouts. of youth problems. Professor William isolating age groups, and letting youth’s Julius Wilson has discussed the outmi- Second, the combination of the lives proceed without much interaction. gration of middle class people from in- growth of single-parent households and The combination is devastating: youth ner cities; the rise of a highly estranged a greater proportion of households simply do not get enough time and at- group of people isolated in poor neigh- where both parents work reduces the tention from adults, and our society borhoods; the changing structure and amount of time for parental guidance poses ever more complex challenges for location of jobs — not to mention the and support. youth to grow up healthy and make the unintended effects of affirmative action transition to adulthood successfully. and civil rights, which may have con- Third, an increasing proportion of tributed to middle class flight, and the poor youth in urban areas are discon- The recent emergence of a strong changing structure of families with the nected from schools and the labor mar- political force which holds that almost associated challenges of single parent- ket and isolated from adults who work. every challenge and problem in an indi- hood and illegitimacy. I think life is too

26 Evaluation Forum n Issue 12 n Summer 1997 Commentary

vidual’s life is individual and private these forces and trends are having neg- Question: What do you feel have been rather than social and public in nature, ative effects on a larger share of all the effects of these trends in the 1990s is also a powerful obstacle to providing youth, regardless of their color or socio- regarding 1) the nature of the Ameri- youth with the supports, direction and economic status. And it is difficult for can workforce in the 1990s, given the opportunities they need. Though this us to recognize that our collective social global economy in which the U.S. must view has the potential to bring about a accountability for helping our youth now compete, and 2) the integration of stronger sense of responsibility for face this world is not being addressed. American youth into that workforce? youth at the individual, family and community level, for many families Alan Zuckerman David Brown and neighborhoods in many areas of Youth are becoming even more iso- Career opportunities for many of our the country that will simply not be suf- lated. Adolescence is lasting longer as nation’s young people (and their par- ficient. Public action will be necessary the transition to work and a stable fam- ents) are seemingly evaporating before also. This political force combines with ily structure occurs later in life. Single- their eyes. To remain competitive in the a genuine concern for public budget parent families and two-wage-earner global economy and to satisfy investors, deficits to further limit the possibility of families often mean that young people American businesses have slashed labor strong public policies toward our youth. lack a constructive relationship with an costs through drastic downsizing, wage adult. This is coupled with rapid The increasing geographical distance and benefits cuts, plant relocations, changes in the economy and work, of poor urban and rural communities and the hiring of contractual, part-time which mean many young people are from areas of job growth further hinders and temporary help. Each of these facing career preparation decisions the ability of families and communities cost-cutting strategies contribute to the without the counsel of a knowledgeable to undertake alone, without public in- uncertainty and despair faced by Amer- adult. tervention, the building of supports and ica’s emerging labor force, its youth. opportunities their youth need. This Lacking information, feeling unwant- The well-paying and relatively secure phenomenon not only limits adults, but ed and seeing a future of short-term low-skill jobs that enabled earlier gener- also presents to youth a formidable, jobs paying poverty wages, many young ations of marginally-educated young limited and depressing view of what people have little hope of getting a Americans to support families, pur- their future prospects are and how good job. Lacking hope, some young chase homes and raise their economic much the larger society cares about people drop out of school and further and social status have virtually disap- them. diminish their opportunities. Leaving peared. school has always hurt one’s chances Our devotion to free markets and The nation’s changing job structure for getting a job, but the reduced in- freedom — our hallmarks and strengths has also contributed to increasing eco- vestment in “second chance” programs as a nation — have backfired regarding nomic inequality based on educational eliminates opportunities for the most at- healthy youth development. Weapons, attainment. The gap between the aver- risk young people. Youth do not vote drugs — and a media devoted to the age wages of the high school graduate and are out of the mainstream. Taxpay- sensational, the negative, the purely and the college graduate has continued er revolts have reduced funding for ed- commercial — these consequences of to grow since the mid-1970s. Public ucation and increased tuition and fees. our freedom have helped shape an en- schools, which adequately prepared Low income young people have less vironment which requires more and the vast number of young people for chance than they had before. The ero- stronger adult and institutional interac- careers in the low-skilled manufactur- sion of opportunity for young people tions with youth, at the exact time that ing jobs generated by an industrial creates competition among youth and such supports are getting weaker. We economy, have not kept pace with the between groups of young people. Black are confused about what to do, since increasing expectations and skill de- and white, Anglo and Hispanic, urban we do not like to limit adult or corpo- mands of the postindustrial economy. and suburban, young women and rate freedoms, and do not have much Secondary education in America is un- young men are all competing for scarce precedent for doing so, even for the apologetically focused on the college- resources — education, jobs, other sup- good of our own youth. So we focus on bound, while the remaining majority port. exhortation, not public action. are too often offered little relevant skills The fact that so many of the youth Finally, the criminal justice system is or preparation that will enable them to most affected by the above trends are becoming the principal youth-serving successfully enter contemporary labor of color, and from poor families, only agency in this country, and it creates a markets. The limited value of the high exacerbates our confusion and unwill- self-fulfilling prophesy. Young people school diploma is demonstrated by the ingness to act. These are the groups we are arrested in even larger numbers and fact that high school dropouts with are quickest to hold individually ac- more of them go to jail. In jail, they are eleven years of schooling do not fare countable for their problems and diffi- offered little or no youth development any worse than their peers who com- culties. We find it difficult to arrive at or workforce development opportunities plete high school or achieve a GED but balanced accountability between the and they become apprentice criminals. do not pursue postsecondary education individual and the larger society. Thus The cycle is repeated for far too many or training. it is hard for us to acknowledge that youth, until they are incarcerated for long terms or finally find an alternative.

Evaluation Forum n Issue 12 n Summer 1997 27 Commentary

At the recent National Education tant culprit for the problems that many warded for their hard work and Summit convened by the National Gov- young people experience. I do know, years of playing by the rules, it is ernors’ Association and IBM in Pali- however, that recent research from becoming increasingly difficult to sades, New York, summit co-chair Lou- Child Trends, as well as Rand’s Jacob convince them that they are wanted is Gestener, Jr., the CEO and Chairman Klerman, show that many young people and necessary in today’s economy of IBM, stated: “In an increasingly glo- can be identified in their twenties as and society. If the rewards are lack- bal economy, I am not liking our (Amer- suffering the long-term consequences ing for them and for their parents, ica’s) chances.” of estrangement or disconnectedness while a smaller and smaller propor- begun in the latter part of high school. tion of society is reaping incredible Indeed, we are already seeing the During the high school years they have gains, rising resentment must surely results of the collapse of our public been out of the workforce and out of result. schools, and all of us — states and school, and this has had rather dramat- businesses alike — are bearing the costs n Declining motivation. Similarly, these ic effects a few years later. On the oth- of failure. Many of the nation’s leaders trends affect youth’s motivation for er hand — and this may confuse read- concur that if the United States is to planning ahead, postponing immedi- ers — even among young people maintain its preeminent position in the ate gratification and investing in disconnected during the high school world economy and in terms of the their futures whether through an ed- years, the really horrific outcomes expe- standard of living enjoyed by its citi- ucation or a series of jobs. Certainly, rienced among young adults in their zens, strategic initiatives need to be as we have seen, not all jobs require twenties is concentrated in those who undertaken jointly by the public and a college diploma, but most jobs have had the most severe problems private sectors that will improve the should hold out the prospect of a liv- during the high school period. Put dif- transition from school to work and en- ing wage and for some chance of ferently, many children — perhaps up able young people to make productive advancement over time. to a third — suffer from a lack of en- contributions to the economy. However, gagement with mainstream institutions. the dominant theme of American edu- But even among the one-third, a small- Gary Walker cation, which asserts that a four-year er group faces incredible trouble as ear- college education is the only reasonable The effects of these trends are now ly as the early twenties. track to economic self-sufficiency and becoming evident: more youth, espe- social status, stands as a persistent bar- I do not think the policy implications cially in poor and working class com- rier to developing additional career of these concentration effects have per- munities, are without the technical and pathways for secondary-school stu- suaded policymakers. Age-appropriate social skills, and the “connections” dents. targeting, prevention beginning with necessary to make a good living. There middle-school age youth, and income is also more youth crime in those areas, of course, and disengagement from Gary Burtless targeting are themes that attract con- siderable ambivalence. Targeting, many conventional values. The most vivid As a result of trends that have ad- believe, is for racial and ethnic minori- way to understand how difficult it is versely affected less-educated labor ties (the “undeserving” poor?) or it pro- for today’s youth to find their way into force entrants, young men (and young duces, in their view, stigma that dam- this economy with some expectation of women with very low levels of school- ages all programs — or perhaps “success” is to look at what middle and ing) earn substantially less than young Americans believe that targeted pro- upper class families are devoting, in people, with similar qualifications, grams are lower quality programs. Sim- terms of their financial resources and earned twenty years ago. The loss in ply put, universalistic programs are connections, to helping their youth pur- earnings has probably delayed the age more palatable to policymakers. This is sue an upward economic path. Adults at which they marry and may have de- certainly the case with the several Con- with good incomes are spending more layed the age at which they first have gressional proposals for a new work- money, and worrying much earlier in children. Adolescence has been ex- force development system for young their children’s lives, to ensure their tended for many young people so that Americans. To summarize, our research economic futures. They send their chil- it now includes the late teens and early is showing that only a minority of the dren to private schools and universities twenties. Young people have been poor suffer really awful long-term ef- at great cost, and also spend a great forced to live with their parents longer, fects, yet our policies are generally deal of money to provide them with the because the job market offers no obvi- turning away from these groups. recreating and other positive supports ous way for them to earn enough mon- they need. If that is the realistic model ey to support themselves. of what it takes to integrate young peo- Chris King ple successfully into the workforce, we Andrew Hahn Several effects should be mentioned: should be very sobered about the pros- pects for a high percentage of Ameri- n Increasing alienation and resentment. I would not want to hypothesize can youth. that the globalization of the workforce As youth/young adults see their par- in the 1990s is the single most impor- ents being punished rather than re-

28 Evaluation Forum n Issue 12 n Summer 1997 Commentary

Alan Zuckerman portunities Program, a multi-site cation services, recreation and other youth development demonstration services for all youth, regardless of Fewer young people are being inte- targeting youth in AFDC families en- their income or educational status. grated into the American workforce un- tering the ninth grade, validated the An evaluation of the original seven til their late twenties, when they are potential of comprehensive programs pilot sites conducted by the Acade- more mature and have higher level that provide sustained (four years my for Educational Development skills. Those lacking education are more plus) and consistent support, prod- found that in six of the seven com- often relegated to long-term low-wage ding, and encouragement. The eval- munities the overall dropout rates jobs with little opportunity to enter a uation found that the participants declined by as much as fifty percent career that pays a living wage. The gap achieved more positive outcomes in one community, and in all six between young and old, rich and poor, than a group of comparable youth. communities for which data were employed and unemployed grows. More available the rate of births to teen- n Mentoring: a recent Public/Private young people are poor and remain in age girls also declined. the secondary labor market until their Ventures study affirmed the efficacy mid-to-late twenties. School-to-Work of mentoring programs. The study has the potential to ease the transition conducted at eight local affiliates of Gary Burtless to employment for some young people, Big Brothers/Big Sisters of America So far, our nation has not discovered but the scale is not yet sufficient to found that mentored youth were less ways to ameliorate the problems of reach its goal of “all youth.” likely to start using drugs and alco- young people who do not flourish in hol; had improved school attendance school. It has not established institu- and performance; had improved atti- tions that are parallel to college but Question: What social policies and tudes towards completing school; that can impart occupational skills to program strategies over the past decade and had improved peer and family youngsters who fail to prosper in the or so do you feel have been the most ef- relationships. The researchers classroom. There may be two promising fective in enhancing the life chances of stressed that the rigid standards and signs for the future, however. Commu- American youth, and in resolving or supports employed by the national nity colleges have increasingly taken on mediating youth education, training Big Brothers/Big Sisters office were the role of preparing young people for and employment problems —based on critical to generating the encourag- specific occupations. Though most two- insights from competent policy analyses ing impacts. For instance, all local year colleges probably emphasize desk and evaluation research? affiliates must adhere to standards work too much, at least they are willing governing the screening and accep- to offer credit courses for the acquisi- David Brown tance of both youth and adults, the tion of occupational skills. training of volunteers, the matching Although many of the recent evalua- process, the frequency of meetings, Another promising sign is the grow- tions of youth programs have been dis- and the ongoing supervision of ing recognition at the state and local couraging, many of the findings should matches. level that the current school-to-work not be surprising given the nature of system is failing our non-college-bound the predominant narrowly-focused, n Contextual/applied learning: the youngsters. Recognizing a problem is short-term and deficit-driven approach- Center for Employment Training’s not the same as creating a solution, but es. Research suggests that short-term contextual learning model has been it is a start toward framing one. programs that focus on improving job proven to be effective in improving search skills or that simply offer remedi- the outcomes of out-of-school youth. Andrew Hahn ation or GED preparation at best have The San Jose, California-based pro- I think that our evaluation of the short-lived success, and they do not gram uses a holistic approach that Ford-funded Quantum Opportunities adequately prepare many youth for suc- integrates basic skill remediation and Program (QOP) should be of keen inter- cess in the labor market. However, job skill learning. CET first identifies est to workforce development special- some recent research has demonstrated jobs that are in demand and then ists because it used a rigorous research that comprehensive and developmentally- builds a training curriculum focused design (random assignment along with oriented strategies can yield more posi- on job-specific skills. Basic skill de- case studies) to track over time the tive outcomes. velopment is embedded in the real- world training curriculum. success and problems experienced by n Community service: research has small groups of young people involved demonstrated that young people n Comprehensiveness: The U.S. De- in an interesting program with several who engage in service to their com- partment of Labor’s Youth Opportu- special design elements. These design munity display higher levels of nities Unlimited demonstrations ex- elements included a financial incentive “prosocial development.” One study plored the potential effectiveness of for students (stipends for program par- found that service participants expe- a holistic service approach that em- ticipation, bonus payments for level of rienced significant decreases in tru- phasized positive youth development effort, and an accrual account for use ancy, deviant social behavior and and was targeted on all youth living in an approved program of post-second- disciplinary problems. in impoverished communities. This ary education); special incentives for demonstration, which was codified staff; and a philosophy of sticking with n Sustained supports: research con- as Youth Fair Chance in 1992, “satu- kids and not thinking of them simply as ducted by the Heller School at Bran- rated” six urban and one rural high- slots. But mostly, what made QOP spe- deis University of the Quantum Op- poverty areas with job training, edu- cial is that it tested a long-duration (9th

Evaluation Forum n Issue 12 n Summer 1997 29 Commentary

grade through 12 grade, with some fol- students throughout high school, thus Youth Conservation Corps reveals posi- low-up), year-round, multi-service ap- creating a more supportive learning en- tive employment-related impacts among proach involving educational remedia- vironment. Adult mentors provided African American and Hispanic youth, tion, exposure to arts and culture, work through Big Brothers/Big Sisters of reduced arrest rates, and a positive experience, community service, life America led to improved school atten- monetary benefit to society. skills and other services. It put the lib- dance and reductions in substance These research findings and best erals’ mantra to the test, namely that abuse. practice lessons need to be disseminat- long-term, age-appropriate services, in Programs have used financial incen- ed more broadly, especially to youth a community context run by caring tives in varying ways. Ohio’s LEAP pro- practitioners, and they need to infiltrate adults, could make a difference. gram uses a package of financial incen- the public conscience to convert public There are few programs like QOP in tives and penalties to induce teen skepticism into public support. Public the United States. Our pilot research parents on welfare to stay in or return policies need to be revised to encour- was encouraging enough in terms of to school. While the overall results are age and reward program operators for net impacts on schooling, jobs, family mixed, the program had positive effects adopting these proven methods for life and other variables, that the Ford on school attendance, high school di- serving young people. Foundation and the U.S. Department of ploma/GED rates, and employment Labor are replicating QOP in a number rates for teens who were in-school Chris King of sites this year. OICA, the communi- when the program began. QOP used fi- ty-based organization started by Rever- nancial incentives tied to hours of par- Those with far greater depth and ex- end Leo Sullivan, continues to provide ticipation and achievement of specified pertise in youth programs and their ef- technical assistance and Mathematica benchmarks. Both of these approaches fectiveness can address this question Policy Research, Inc. is the evaluator. have rewarded young people who have more fully than I. The evaluation record “played by the rules”. for youth interventions over the past twenty-five years or so is generally a Rob Ivry Many successful programs feature disappointing one (Betsey, Papagoreiu While the body of evidence on the paid work experience offered in combi- and Hollister, 1980, and more recent re- effectiveness of youth programs is gen- nation with education, or provide views). Nor was the recent JTPA evalu- erally discouraging, there have been hands-on experiential training in occu- ation encouraging in terms of signifi- some successes, and the research find- pational areas experiencing job growth. cant net impacts on employment and ings and best practice lessons provide a The Youth Incentive Entitlement Demon- earnings for youth (Bloom et al., 1993/ foundation on which to build. The re- stration, which operated in the 1970s, 1994). Many of the MDRC reports on a sults tend to be more encouraging for was the most ambitious youth demon- variety of youth employment and train- in-school youth than for youth who stration ever undertaken. Economically ing initiatives in recent years offer little have dropped out of school. Successful disadvantaged youth, ages 16-19, in the to be excited about. And there is con- programs tend to have the following seventeen selected communities were siderable debate at present as to the core elements in common: the presence guaranteed jobs (part-time during the value of “job shopping” or, conversely, of caring adults, financial incentives, school year and full-time during the the value of a more structured transi- hands-on training centered on work summer) on the condition that they re- tion from school to work (Heckman et with a strong job development and mained in school (or returned if they al, 1995; Marshall, 1995). placement component, and opportuni- had dropped out) and met specified at- Yet two strategies or programs stand ties for peer support and leadership de- tendance and performance standards. out. First, evaluations of the Job Corps velopment. The demonstration, which enrolled over 76,000 youth, virtually eliminated the (not coincidentally, one of our most ex- These elements manifest themselves disparity between white youth unem- pensive and intensive investments in in different ways. Here are some exam- ployment rates and the unemployment hard-to-serve youth) by Mallar et al. ples. Caring adults can be teachers, rates of students of color. (Mathematica Policy Research) and mentors, case managers, counselors or others have demonstrated its effective- supervisors. An adult presence is the In YouthBuild, young people are ness along several key measures. Note, scaffolding that permits young people trained in construction skills, receive too, that even with such findings in to grow, experiment, regress safely, paid work experience renovating vacant hand, Congress and prior Republican take risks, develop standards and val- housing for low income families, re- administrations have tried hard, time ues, build networks and transition to ceive GED preparation, and participate and time again, to ‘do in’ the Job adulthood. Sustained adult contact over in peer support and leadership develop- Corps, demonstrating that even success the four years of high school is the cen- ment activities. The Center for Employ- may not be sufficient to preserve fund- terpiece of the Quantam Opportunity ment Training’s excellent track record is ing efforts on behalf of youth. Program which produced large impacts attributable to short-term training cours- Second, going even farther back, the on everything from school completion to es which integrate basic academic skills Perry Preschool and Head Start evalua- pregnancy prevention. In Career Acade- instruction with vocational training, a tion findings suggest that if we are to mies, which restructure comprehensive pervasive focus on employment, the make serious headway for youth, high schools into schools-within-schools use of hands-on instructional practices, broadly defined, we might do well to organized around a career theme, and the involvement of employers with concentrate even more of our resources teachers stay with the same group of all facets of training. A recent study of

30 Evaluation Forum n Issue 12 n Summer 1997 Commentary

on preschool education and early child- Alan Zuckerman necting acquisition of knowledge hood development. By the time chil- and competencies to work and ca- dren have reached age four or five, We have learned a great deal in the reers. many of their crucial developmental past thirty years about the effectiveness n High standards: young people and patterns have been well established, of programs that have an impact on the organizations that serve them for better or worse. For far too many, young people. Programs like School-to- must be judged against very high the remainder of their education may Work, Center for Employment Training, standards of performance to assure be spent attempting to reverse the ef- YouthBuild, and Conservation Corps that young people meet or exceed fects of poor beginnings. have made a significant contribution. However, there is no single policy or the highest academic and employer program model that will transform standards. Gary Walker young people into productive, self-suffi- Our public policies for helping youth cient adults. develop and transition to adult roles What are your recommen- NYEC has launched the Promising Question: have been and are generally weak. dations for future directions in social and Effective Practices Network (PEP- Public policy has always largely ignored policy and programming for youth, giv- Net) to create higher standards for adolescents and early to mid-teens. Be- en the problems, successful efforts and youth employment programs, based on ginning with the passage of the Job possibilities you have identified? the foundation of successful youth em- Training Partnership Act (JTPA) in the ployment/development programs, prac- early 1980s, we have had no effective titioner experience and research. PEP- David Brown public policy for older youth who need Net will establish high standards for Researchers and leading youth ser- assistance to enter the labor market. effective programs, invite nominations vice practitioners alike are increasingly However, there have been a number and recognize programs that meet these promoting and adopting an asset-driven of programmatic strategies that have standards. The standards fall into three approach for addressing youth prob- had promising results. P/PV’s recent broad areas: youth development, work- lems. They concur that most children evaluation of Big Brothers/Big Sisters force development, and quality man- and adolescents have basic develop- shows what can be done with early agement. All three areas are essential mental needs that are frequently ig- teens. The Quantum Opportunities Pro- for quality programming. Some of the nored by the community institutions gram in Philadelphia also indicates that elements of effective programs are: charged with serving them. To promote it is possible to dramatically increase the development of comprehensive n Education: youth must be supported youths’ success and decrease their seri- youth development programs, commu- and encouraged to stay in school, go ous problems over the course of the nities should collaboratively fashion a back to school, or find an alternative high school years. The Center for Em- continuum of developmentally appropri- school. All young people should be ployment Training has had good results ate services and activities that build the entitled to funds for their education with young adults. All three of these social, moral, emotional, physical, voca- until they achieve a core set of com- programs put an emphasis on establish- tional and cognitive capacities of young petencies. ing strong adult/youth relationships as people. In particular, activities should a centerpiece to their programming. All n Youth are a resource: we must invest be designed that promote responsibility three focus on providing early positive in the minds and bodies of young and self-reliance, foster positive peer experiences, strong feedback and ongo- people. They need basic reading, interactions, and ameliorate the nega- ing support even after the initial posi- writing, math, problem solving, in- tive influences of adolescent peer tive experiences. All three provide terpersonal and work-related compe- groups. In addition, as job security has lengthy and consistent structure to the tencies. This will require a high de- become a relic of a bygone era, young young person’s life. In short, they pay gree of collaboration, individualized people must be encouraged to pursue attention to the developmental aspects plans, and involving young people in lifelong learning opportunities to remain of youth, and do not view young peo- their own development. economically competitive. ple solely through the prism of their de- n Community service: young people re- Principles for policymakers to con- ficiencies. spond to opportunities to improve sider, based on a small meeting of The hopeful aspect of these common their community. Community service many national youth policy and pro- features of effective programming is combines altruism and work experi- gram experts that was convened by the that there is nothing very mysterious ence. National Governors’ Association in Octo- about them. They are doable. The dis- ber, 1995, include the following: n Support from caring, informed couraging aspect is that they require a adults: research and experience doc- n Provide resources to enable commu- sustained commitment to young people ument the importance of a positive nities to provide nurturing and safe as developing creatures with both relationship with a caring adult — places for young people. strengths and weaknesses, whose best family, mentor, teacher, counselor or chance for avoiding serious problems n Target scarce resources on disadvan- neighbor. lies as much in nurturing their taged communities and make servic- strengths as it does in focusing on their n Integration of work-based and aca- es accessible to all youth in the weaknesses. But our approach to using demic competencies: developmental community, rather than stigmatize public policy is that it is a last resort for plans that include applied learning youth by labeling them as at-risk, fixing problems and weaknesses. And combined with the development of basic skills deficient, teen parent, we want solutions quickly. basic educational competencies mo- delinquent, etc. tivate young people to learn by con-

Evaluation Forum n Issue 12 n Summer 1997 31 Commentary

n Build on the strengths of youth prenticeship, classroom training, cooper- Research also leads to the impor- (rather than focus on their perceived ative education, school-to-apprenticeship tance of strengthening the connections deficits), and positively channel training, school-based enterprises, and between school, basic skills, work and their energy and enthusiasm, recog- career academies. Within each model, the workplace. For this reason, I fully nizing and exploiting the power of schools and employers may take differ- endorse the goals of the national the adolescent peer group. ent approaches toward training and re- School-to-Work Opportunities Act, but warding participants for the work they I also have no illusions about its effec- n Empower youth by involving them in do. Some programs emphasize the in- tiveness as an anti-poverty strategy. For governance and decision making and volvement of adult mentors for each that we have to think more imagina- promote citizenship and responsibili- trainee. Others emphasize classroom in- tively about involving community ty through community service. struction in schools. Others may require groups in school-based workforce de- n Coordinate or integrate the full range students to spend a large proportion of velopment, helping these groups to de- of youth services within the commu- the academic year in a job — say, fif- velop their capacity, integrating “hard nity, including recreation and health teen or more hours a week — while and soft” skills training, and targeting services. other programs offer no pay to trainees national workforce policies to either who work. Experience has shown that neighborhoods or people who need the n Link to the region’s economic devel- one of the toughest challenges facing help most. opment strategies, to ensure that workplace-centered learning is recruit- youth are exposed and connected to Research also recommends the im- ment and retention of local employers real employment opportunities. portance of increasing the intensity of to participate in school-to-work pro- services. This is why I addressed the n Provide contextual/applied learning grams. Some school systems may devel- importance of the Quantum Opportuni- opportunities and integrate work- op innovative methods to attract and ties Program in a preceding question, based and academic learning. keep employer participation. and that is why I am enthusiastic n Provide consistent and credible adult The variety of program models that about the work of the International mentors and role models. school systems and localities have de- Center for Residential Education, a veloped opens rich possibilities for a Washington DC-based group that is try- n Establish high standards for all sound evaluation of alternative ap- ing to help communities establish resi- youth and incentives to encourage proaches. Investigators can attempt to dential education options that avoid the the achievement of competencies determine whether one program model stigma of the institutional settings of and goals. has been consistently more successful the past. n Provide sustained services and sup- than others. They can try to estimate I also believe that policymakers and ports. the relative effectiveness of small varia- community planners need to think more tions in treatments within the same ba- about managed care tools that can sic model. At the moment, analysts and Gary Burtless track clients through an array of servic- policymakers have little reliable infor- es in the community. And I would like Over the next decade, the main goal mation to show whether one approach to see new linkages between communi- for sensible policy is to develop effec- to workplace-based education is better ty development and school-to-work ini- tive program models to impart skills to than another. Since many local pro- tiatives, in order to promote longer- the young people who will not enter or grams are now in their infancy, pro- term, more intensive services. I think complete college. Competent evaluators gram administrators have to learn about intensity could be promoted by more should attempt to learn whether certain the relative effectiveness of different use of youth entrepreneurship pro- school-to-work program models yield approaches. For that reason, there is grams. These combine traditional class- better results than others. In the long likely to be a large payoff from research room training with hands-on experien- run, this kind of information will have that reliably distinguishes between suc- tial learning through business enormous value to policymakers and to cessful and unsuccessful approaches, development. Students stay in contact young people trying to get established as well as from research that shows with these programs longer than might in the job market. At the moment we whether certain kinds of students and be the case with regular youth pro- know very little about the long-term dropouts can benefit from particular grams, because the students need ad- benefits of a reformed school-to-work school-to-work models. An important vice and assistance from the program system. Students, employers, and focus of a good evaluation should be staff to help maintain the small busi- school administrators have too little ex- measurement of the relative effective- ness enterprises. perience under recently developed pro- ness of different program models. grams for us to draw reliable conclu- Finally, research suggests that we sions about the longer-term effects of Andrew Hahn must promote effectiveness in what mature programs that are run by expe- will be a decentralized structure. This rienced educators and employers. How- Research supports the recommenda- translates into local monitoring and ever, careful analysis of results from a tion to focus on youth as youth, some- evaluation of workforce development variety of good programs can shed light thing that was often lost as the work- programs, more capacity building, and on whether certain program models force field rushed to create specialized a greater role for university-community show special promise. niches. So, for example, resume writing partnerships. It also speaks to involving workshops may not be appropriate for The Office of Technology Assessment all sectors of the community in work- fourteen-year-olds, but higher educa- has identified six broad models of work- force development activities, including tion awareness programming is terribly place-centered learning — youth ap- employer groups — although we have important for this age group. few good models of employer-side inter-

32 Evaluation Forum n Issue 12 n Summer 1997 Commentary

ventions with implications for deep ur- making “average daily attendance” sponse was not resignation but rather a ban poverty. funds portable (the funding follows redoubling of our efforts until we be- the student) and by broadening the came the first to put a man on the Rob Ivry use of criminal justice funding for moon in 1969, only a bit more than a preventative activities. decade later. We need to approach our We need a national policy of ade- experience with youth initiatives in a n Continues to encourage summer ed- quate and sustained investment in similar fashion. young people. The negative youth re- ucational activities to be offered in sults from the national evaluation of the conjunction with summer employ- Second, we must begin to restore Job Training Partnership Act reveals the ment. the employer/employee compact that flawed strategy of trying to shoehorn existed in this country for most of the n Provides strong support for new am- period from the late 1930s to the 1980s. youth into an adult system, with per- bitious demonstrations to expand the This compact was never as strong as it formance standards that work at cross- knowledge base concerning what was in our trading partners in Western purposes to the developmental needs of works for youth — for example, Europe or Japan, but it was essential to young people. The federal investment demonstrations which challenge the the fabric of the nation and our market- in youth programs has been cut back in existing paradigm of education as a based system. The current trend, in recent years, and the eventual passage prerequisite for employment, which the community of employers of a workforce development block grant through alternative strategies which views its primary and possibly its sole could further infringe on youth funding provide skill building and education obligation as being to its shareholders and services. Public policy needs to activities on the back end of em- to the exclusion of its employees — recognize the wisdom of a youth in- ployment once students start work- whether concerning the provision of vestment policy with the underlying ing. premise of “building on strengths” rath- equitable wages, health and related er than “remedying deficits” and which There also needs to be a recognition benefits associated with employment, incorporates the following principles: that solving the youth problem and ex- and some sense of security — must be panding opportunities for young people reversed. n Implements a funding structure that will not come cheaply and the results Third, we should contemplate targets funds where they are needed will not occur overnight. There needs to adopting capital budgeting nationally most and provides strong financial be sustained investment and public ex- and in the states, encompassing both incentives for states and localities to pectations need to be realistic about human capital as well as infrastructure. adopt and tailor proven and promis- what can be achieved in the short- Why should the financing for education ing program approaches such as term. The future health and vitality of and training, which Dennison (1962, those described above. our country hinges on the extent to 1991), Harbeson (1973) and others which we adopt the policy tools to en- n Adopts a flexible system of perfor- have demonstrated to be the dominant able the current generation of young mance standards which acknowl- factor in explaining our nation’s eco- people to succeed in school and the la- edges that the youth maturation pro- nomic growth over time, be handled on bor market, and successfully make the cess varies by individual, and a current accounts basis just as the transition to adulthood. rewards progress toward education- staffing of prisons or income mainte- al, employment and personal devel- nance, when it is more appropriately opment outcomes. Chris King viewed and financed as an investment? n Strengthens families and communi- I will offer a number of broad social It is also time to address the fact that ties to provide greater support to policy and specific program recommen- we have increasingly pushed the bur- youth and increases the engagement dations, beginning with the former. den of financing postsecondary educa- of young people in productive activi- First, we truly must begin to view all tion towards individuals and families, ties after school hours. children as our nation’s future and while the tax code offers them little bring our social policies around accord- support. We must open the door to n Authorizes career preparation and ingly. We cannot afford to ‘write off’ some form of individual training ac- paid work experience as fundamen- any child. In crass terms, we all need counts or related measures with prefer- tal activities within a youth develop- them to help support us — physically, ential tax treatment. ment strategy in conjunction with emotionally and financially — in our education, leadership development, More specifically, there are several dotage! More importantly, it is a moral and skill-building activities. other actions we should pursue: imperative, especially in a nation with n Extends the core elements of the the wealth we possess. We cannot af- n The federal and state governments school-to-work movement — school- ford to punish children for the per- should encourage employers to pur- based learning, work-based learning, ceived ill behavior of their parents by sue modes of high performance work and activities which connect the consigning them to another generation organization (e.g., team manage- two — to an overall youth develop- of poverty and insecurity. Each youth ment, shifting work responsibilities ment strategy. in this country must be viewed as a to ever lower levels, pay based in valuable asset not to be squandered. I part on group effort, flexible produc- n Supports creative funding strategies am reminded that when our attempts tion, the use of the latest technolo- for financing alternative education to launch satellites into space failed gies — Marshall, 1992; Batt and Ap- and employment programs for out- miserably in the late 1950s, our re- plebaum, 1994; Osterman, 1995). of-school youth by, for example,

Evaluation Forum n Issue 12 n Summer 1997 33 Commentary

This should include the adoption of policies in the U.S. Note that several framework requires a bit more hard one of the centerpieces of the ISO evaluations have shown that partici- thinking and a lot less rhetoric than is 9000 process, namely fostering tight pants in these programs see access- typical for the formation and execution and lasting producer/supplier rela- ing quality child care as one of the of public policy. tionships, in this instance those be- major benefits of participation (King tween employers and schools as sup- et al., 1994; MDRC, 1996). Alan Zuckerman pliers. To become ISO-certified, I would propose the following rec- which is increasingly a prerequisite Gary Walker for conducting business in Western ommendations: The first thing we will have to do is Europe and elsewhere around the n Maintain a federal commitment to at globe, a firm must demonstrate that to accept public policy as a tool for improving the basic supports and op- risk youth programs: There must be it has cultivated such supplier rela- adequate funding for both summer tionships and that it ensures that portunities that youth need for healthy development, and not see it as a neces- and year-round programs. Rational these suppliers are trained to the discussion about the merits of in- same standards and expectations. sary evil that only gets used when problems get out of hand. The recent vesting in youth is vital to the future Were employers to view schools as of this country. one of its primary suppliers, not just School-to-Work Opportunities Act, and becoming involved but rather truly the proposed Youth Development Block n Focus on special needs of urban and engaging schools as an integral part Grant, are good signs that some policy- rural youth: Our cities cannot be of their production/service process, makers are ready to make that shift. forgotten and abandoned, and youth would face far better and The second necessary change is that young people must be connected to more structured employment oppor- we must accept that it takes resources education and jobs. There must be a tunities. and time — in short, genuine public revitalized urban policy. Funding commitment — to accomplish sustained must be targeted to cities and poor n The Commission on the Skills of and durable positive development in rural areas where the needs are America’s Workforce’s America’s greatest. Choice report (1990) contained a young people. Neither Big Brothers/Big number of recommendations on Sisters nor CET developed its organiza- n Accountability: Demand that youth youth policy which should be seri- tional strengths overnight. The need for programs meet rigorous standards. ously considered for adoption. patience in building organizational ca- Young people must be prepared to Among these are use of the Certifi- pacity is especially important in helping meet world-class standards. cate of Initial Mastery and others. youth connect with the labor market, since that requires developing good n Invest in building the capacity of community-based programs and staff: n Nationally, Congress and the Admin- working relationships with employers. istration should pursue full and con- The uncertainty of federal funding tinuing funding of the School-to- The good news is that the program and the transition to “block grants” Work Opportunities Act of 1994. As strategies that have been effective is placing many experienced and things now stand, the federal gov- could be taken to scale without deci- proven community-based organiza- ernment will provide several years of mating the federal budget. For exam- tions in jeopardy. Quality services re- “seed money” after which time it is ple, our estimate is that a $3.0 to $4.0 quire skilled staff and stable organi- up to financially-strapped states and billion commitment of public resources zations. would provide intensive mentoring for local governments to pick up the tab n Jobs for at risk youth: Young people and continue the work. This is far all young adolescents who need it. Of course, it would take several years to enroll in job training programs be- too important an effort for the nation cause they want to get a job, earn as a whole to let it simply devolve to build the institutional capacity to spend that money effectively. money and support themselves. Of- subnational levels of government. As ten entry-level jobs are scarce, pay we have seen too often, some states Assuming we can achieve the little money, and may be dead-end. will do well with this responsibility change in approach and increase in re- Central cities and remote urban ar- while others are likely to fail misera- sources and time commitment neces- eas have more job seekers than jobs. bly. We cannot risk this outcome. sary, we could speed the process of Job creation, including publicly-sup- achieving widespread effectiveness if ported jobs, and the development of n Finally, although it may be beyond the task at hand, rather than ad- we would drop the tired polarity of career pathways are essential if at- dressing child care as a support ser- “central cookie cutter” vs. “all truth is risk young people are to find jobs vices issue in our many E&T pro- local” and administer and monitor pub- that enable them to support them- grams, we must see it as yet another lic funds on the more realistic basis selves. opportunity for investment. Too that “both ideas are true.” There are core principles of effective program- n Young people are a resource, not a many states, not to mention the problem: Investments in youth em- Congress, see funding for child care ming that must be adhered to in every situation — and each locality does have ployment and youth development for welfare caretakers, Food Stamp are a more effective and less costly recipients or job training participants different conditions and opportunities within which to carry out those core way to reduce crime and drug solely as a cost to be avoided, when abuse. in fact it is an opportunity to provide principles. In short, there are uniform high quality, early childhood devel- elements to effective programming, and opment services for these children. still plenty of room for local flexibility. This applies also to our work- family To administer public policy in this

34 Evaluation Forum n Issue 12 n Summer 1997 Commentary

vate solution, where the notion of a ment, policies will take shape from the Comments on the commonwealth and a public interest bottom up. Space will increase for com- has dimmed to a faint glimmer. munities to experiment, to agitate and Panel’s Responses to mobilize. All of this presents new It seemed to me that what the pan- challenges and new opportunities for elists are depicting is the emergence of youth research. a system of neo-apartheid. This really is the substance of their findings: a grow- The challenge is to revise our role Susan Curnan ing division between us and them, a and expand our repertoire so that we cordoning off by geography, class and can be more effective and trusted allies with the assistance of race of those groups whose thinking, be- with community change agents. This havior, and performance earn them out- does not mean signing on as booster, or Paul Aaron cast status, and even more so when abandoning our commitment to analytic Senior Research Associate these groups challenge majority stan- rigor. It does mean learning to negotiate Heller Graduate School dards for their well being and insist on relations which are dialogic and recip- Brandeis University having a say. But there seems to me a rocal. It does mean respecting different curious disparity between the central voices, different ways of knowing. It diagnosis, which is grave, and pro- does mean demystifying and democra- There is much food for thought in posed treatments, which are mild. tizing, and disseminating expertise. the responses of the “expert panel.” Its members include colleagues whose The warning signs are as plain as The opportunity is to break down work I know and respect. Many have the handwriting on the wall. Yet the the intellectual apartheid that has long helped define the field in which we all scope and depth of our policies and separated the academy from the com- operate, mapping out the overlapping program recommendations appear munity. We see a society more and boundaries between youth develop- hopelessly incommensurate with the more fragmented and polarized. The ment, education and vocational/occu- underlying problems they are intended collaborative content of our work pational training. to address. The principles and ap- should reflect our dedication to a proaches which panelists promote are wholesome and egalitarian polis. We Especially when it comes to old fine as far as they go. There is genuine should practice what we preach. friends, I balk at trying to reduce di- worth in “asset-driven” strategies; in verse, collective insights so that they fit flexible, developmentally appropriate into a nutshell. Rather than having the performance standards; in a deeper Editor’s Note last word, let me suggest implications knowledge base; and in the intensified and play off both what was said and service delivery embodied in models what was left unsaid. An emphasis on young adults’ need like QOP. But again, I am struck by the for education, skills, training and em- As I read their social diagnoses, the disconnect between the modesty of ployment opportunities is a consistent panelists take a grim view of the cur- these proposals and the sense of loom- message in the panel’s responses. An- rent environment within which youth ing crisis that many of us feel. drew Sum’s research contributes much come of age and are integrated into the I think many of us are searching for to that perspective. His thinking, based workforce. Words like frustration, bleak- ways to reduce this disconnect. It is on extensive research, is revealed ness, insecurity, despair, and resentment hard going. The culture into which we clearly in “Young Workers, Young Fami- recur as a descriptive motif for the pre- have been socialized places a premium lies, and Child Poverty,” a chapter by vailing mood. For more and more Amer- on analytic detachment. We are profes- Sum, Clifford Johnson, and Neal Fogg icans, the opportunity structure is col- sional outsiders, trained and rewarded in Garth and Stephen Mangum’s new lapsing. Access and participation in to observe, measure and report on edited book, Of Heart and Mind: Social effective education and training pro- what others are doing to bring about Policy Essays in Honor of Sar A. Levi- grams are limited to the most prepared. social or institutional improvement. As tan. Sum and his colleagues define the De-industrialization has cut the rungs individual citizens we may choose to youth problems demanding change, out of the ladder to a middle class life become engaged as advocates and ac- identify the interrelated and complex for most Americans. Global forces are tivists. But as “experts” in the field we causes of these problems, and suggest causing the labor market to buckle and are expected to keep at a safe remove change strategies. shift. Demands increase for high-end from the subjects of our inquiry. We are skill sets, thereby widening the gap be- In Sum’s view, a significant propor- taught that distance protects the credi- tion of American youth are experienc- tween education haves and have nots. bility of our findings, and hence the in- The knowledge industry and informa- ing severe labor market problems due fluence that we might bring to bear on to a lack of education, training, and tion age promise no end to these high- the policy process. end demands. work experience, jeopardizing their There was a time when we proceed- own futures and the future of the Amer- In this rapidly evolving two-tiered ed on the assumption that research and ican economy. He points to a wealth of society, winners take all, and losers be- demonstration activities would help data pointing to the sharp decline over come objects of indifference or aver- federal policymakers make wise choic- the past two decades in the inflation- sion, viewed either as Darwinian misfits es. But devolution has shifted the cen- adjusted weekly and annual earnings or as moral pariahs who have no one ter of political gravity from the national of young adults, despite renewed job but themselves to blame. The rise of to the local. This change has enormous growth. Youth, he claims, have been the libertarian right has helped create a implications. In the current environ- adversely affected by the transforma- politics where every solution is a pri- tion of the U.S. wage structure — with

Evaluation Forum n Issue 12 n Summer 1997 35 Commentary

resulting difficulties in securing full- ______. (with M. Forcier) “The Impact ______. Human Resource Investment time, year-round employment, and with of Employment and Training Programs Councils: Summary of State Efforts, the growing incidence of child poverty. on Work Attitudes of Disadvantaged NGA 1994. Youth: A Synthesis of Theory and Evi- Such trends, he contends, have seri- Burtless, G.T. “Current Proposals for dence,” in Youth Employability: Five ous negative consequences for youth School Reform: An Economist’s Assess- Award Winning Monographs. National ment,” in Economic Consequences of and society. Sum’s recommendations Center for Research on Vocational Edu- American Education edited by A. P. for change fall heavily on education, cation. Columbus, OH: Ohio State Uni- O’Brien and R. J. Thornton. Greenwich, skills training, and the expansion and versity, 1982. diversification of job opportunities. CT: JAI Press, 1993. Ivry, R. (with Alan Weisberg) The 22% Solu- ______, M.N. Baily and R. E. Litan. For more detail about the thinking of tion: LEA/CETA in Youth Employment and Growth with Equity: Economic Policy- the members of the expert panel, Training Programs. Washington, DC: making for the Next Century. Washing- please see the following references. U.S. Office of Education, 1980. ton, D.C.: The Brookings Institution, ______. (with Carl Wolfhagen) 1993. Publications in the 1980s Strengthening the School-to-Work Link- ______. “Meeting the Skill Demands age in Youth Incentive Entitlement Pilot of the New Economy,” in Labor Markets, Burtless, G.T. “The Impact of Welfare Re- Projects. New York, NY: MDRC, 1982. form on Recipient Employment, Earn- Employment Policy, and Job Creation ings, and Income,” in Welfare Policy for King, C. T. (with V. Geraci) Employment and edited by L. C. Solmon and A. R. Leve- the 1990s edited by D. Ellwood and P. H. Training Program Performance: Long- son. San Francisco, CA: Westview Cottingham. Cambridge, MA: Harvard Term Earnings Effects and Short-Term Press, 1994. Indicators. Working Paper. Center for the University Press, 1989. ______. “Rising Wage Inequality and Study of Human Resources. Austin, TX: the Future of Work in America,” in Wid- ______. “Work Programs in Welfare University of Texas, 1981. and the Difference They Make,” in Re- ening Earnings Inequality edited by J. forming Welfare: Lessons, Limits, and ______and D. T. Schexnayder. Wel- Norwood. Washington, D.C.: Urban Insti- Choices edited by R. Coughlin. Albu- fare Dynamics in Texas: An Exploratory tute Press, 1994. querque, NM: University of Mexico Analysis of AFDC Turnover and Pro- Charner, I. School Reform and Youth Transi- Press, 1989. gram Participation. Austin, TX: Center for tion. Washington, D.C.: Academy for Ed- the Study of Human Resources, 1988. Charner, I. (With P. Barton et al.) Worklife ucational Development, 1993. Transitions: The Learning Connection. ______. (editor) Coordinating ETA-Ad- Curnan, S. P. (with Policy Research Associ- McGraw-Hill, 1982. ministered Programs: Issues and Rec- ates) Evaluation of the Educational ommendations. Washington, D.C.: U.S. Component of the Summer Youth Em- ______. (With B. Fraser) Youth and Department of Labor, 1989. Work: What We Know, What We Don’t ployment and Training Program. Wash- Know, What We Need to Know. Wash- Zuckerman, A. Establishing a Cooperative, ington, D.C.: U. S. Department of Labor, ington, D.C.: William T. Grant Founda- Interactive Employment and Training 1995. Network. Paper for Florida Employment tion’s Commission on Work, Family and ______In Support of the Youth Devel- and Training Association, 1984. Citizenship, 1987. opment and School-to-Career Move- ______. “Employability Credentials: A ______. “Two Decades and Two Deca- ments and Blurring the Lines Between Key to Successful Youth Transition to logues of Employment and Training,” in What Is Work and What Is Learning. Pa- work,” in Journal of Career Develop- ADHERENT, Volume 4, Winter 1983. per commissioned as part of the Nation- ment, 15, 1988. ______. Individualized Service Deliv- al Youth Research Project. Washington, D.C: U.S. Department of Labor, 1995. Hahn, A. B. (with J. Danzberger) Dropouts ery. Paper for National Governors’ Asso- in America. Washington, D.C.: Institute ciation, 1985. ______. Youth Fair Chance: A Working for Educational Leadership, 1989. ______. At Your Fingertips. Paper for Model for a New Way of Thinking and A New Way of Doing. Conference paper. ______. (with R. Lerman). What Works National Commission for Employment Policy, 1985. Washington, DC: National Convening of in Youth Employment Policy? How to Youth Fair Chance Demonstration, 1995. Help Young Workers from Poor Families. Washington, D.C.: National Planning As- Publications in the 1990s ______(with T. Melchior) Moving To- sociation, 1985. ward Quality Programs: Defining Criteria Brown, David E. “The Case for Second for Quality Program Designs Based on ______. (with R. Sheets, R. Lerman Chance Programs for Out-of-School Lessons from Research and Experi- and E. Butler) Who Will Train and Edu- Youth,” in Labor Notes, NGA, October ence. Commissioned report on behalf of cate Tomorrow’s Workers? The Financ- 1992. U.S. Department of Labor. Washington, ing of Non-College Bound Young Work- ______. A New Paradigm for Youth D.C.: National Youth Employment Coali- ers’ Recurrent Education. Commission tion, 1994. on Work, Family and Citizenship. Wash- Employment and Training Programs, Na- ington, D.C.: W. T. Grant Foundation, tional Governors Association, 1993. ______(with H. Bruno). Measuring the 1988. ______. The Role of Job Training Part- Effectiveness of Workforce Preparation nership Act Programs in School-to-Work Policies at the State Level. Commis- ______. “Job Creation Policies: Do Transition, NGA, 1994. sioned paper. Washington, D.C.: National They Stimulate Demand for Disadvan- Governors’ Association, 1994. taged Young Workers?” in Portfolio: In- ternational Economic Perspectives. Washington, D.C.: U.S. Information Agency, 1987.

36 Evaluation Forum n Issue 12 n Summer 1997 Commentary

______Charting a Course for Excel- ______. “Federal Policy Changes to ______and D. Friedlander. Five Years lence in Youth Development, Education Enhance State and Local Coordination.” After: The Long-Term Effects of Welfare- and Employment Programs. Commis- Chapter 5 in Welfare System Reform: to-Work Programs. New York, NY: Rus- sioned report. Princeton, NJ: Educational Coordinating Federal, State and Local sell Sage, 1995. Testing Service, 1994. Public Assistance Programs, edited by R. Blank et al.” A Primer on Welfare Re- E. T. Jennings and N. Zank. New York, Hahn, A. B. America’s Middle Child: Making form,” in Looking Before We Leap: Social NY: Greenwood Press, 1993. Age Count in the Development of a Na- Science and Welfare Reform edited by tional Youth Policy. Center for Human Re- ______. et al. Texas JOBS Evaluation. R. K. Weaver and W. T. Dickens. Wash- sources. Waltham, MA: Brandeis Univer- Austin, TX: Center for the Study of Hu- ington, D.C.: The Brookings Institution, sity, 1995. man Resources, University of Texas, 1995. ______. “National Human Resource 1994. ______. “Public Spending on the Poor: Policy for Disadvantaged Youth” in Di- ______, R. McPherson and L. O. Law- Historical Trends and Economic Limits,” lemmas in Youth Employment and Train- son. Welfare in Texas: Selected Findings in Confronting Poverty: Prescriptions for ing Policy: Lessons from the Youth Re- and Implications. Austin, TX: Center for Change edited by S. Danziger and D. search and Technical Assistance the Study of Human Resources, 1994. Weinberg. Cambridge, MA: Harvard Uni- Project - Volume 2. Washington, D.C.: ______et al. Engaging High-Perform- versity Press, 1995. U.S. Department of Labor, 1995. ing Employers in School-To-Work Pro- ______. “International Trade and the ______. “Toward a National Youth De- grams. Austin, TX: Center for the Study Rise in Earnings Inequality,” in Journal of velopment Policy for Young African of Human Resources, University of Tex- Economic Literature, June 1995. American Males,” in Nurturing Young as, 1994. Charner, I. “Reforms of the School-to-Work Black Males edited by R. Mincy. Wash- Wills, Joan (contributor). America’s Choice: Transition: Findings, Implications, and ington, D.C.: The Urban Institute Press, High Skills or Low Wages! Washington, Challenges,” in Phi Delta Kappan, Sep- 1994. D.C.: Commission on the Skills of the tember 1995. ______. “The Mission and Structure American Workforce, 1990. ______. School-to-Work Opportunities: of National Human Resource Policy for ______(co-author). School Lessons, Prospects and Challenges. Handbook of Disadvantaged Youth: A Synthesis with Work Lessons: Recruiting and Sustain- Sociology, Section on Education. ? : Recommendations,” in Dilemmas in ing Employers’ Involvement in School-to- Ablex Publishing, 1996. Youth Employment Programming. Wash- Work Programs. Center for Workforce ______(with S. Zeldin). School-to- ington, D.C.: U.S. Department of Labor, Development. Washington, D.C.: Institute 1992. Work Opportunities Through the Lens of for Educational Leadership, 1994. Youth Development. Washington, D.C.: ______. “After the Report: Recommen- ______(contributor). Overview of Skill Academy for Educational Development, dations for Building on the Accomplish- Standards Systems in Education and In- 1996. ments of National Commissions on Chil- dustry: Systems in the U.S. and Abroad. Ivry, R. MDRC’s Employment and Training dren and Youth,” in Building Momentum: Center for Workforce Development. Studies: Implications for Block Grants. Research Strategies to Foster an Effec- Washington, D.C.: Institute for Educa- Briefing paper for U.S. Congress, 1995. tive School-to-Employment Transition tional Leadership, 1994. System. Commission on Work, Family Wills, Joan (contributor). Community-Wide and Citizenship. Washington, D.C.: W. T. Zuckerman, A. (editor) Making Sense of Systems for Learning. Center for Work- Grant Foundation, 1991. Federal Job Training Policy. W. T. Grant force Development. Washington, D.C.: Foundation Commission on Work, Fami- Ivry, R. (With Edward Pauly). “Learning Op- Institute for Educational Leadership, ly and Citizenship, and the National portunities from Two MDRC School-to- 1995. Youth Coalition, 1992. Work Transition Projects,” in Labor Zuckerman, A. (editor) Making Sense of Notes. National Governors’ Association, ______. Toward a National Youth De- Federal Job Training Policy. Volume II. 1993. velopment System. Washington, D.C.: Washington, D.C.: National Youth Em- National Youth Employment Coalition, ployment Coalition and American Youth ______. Emerging Lessons from the 1994. School-to-Work Study. Paper presented Policy Forum, 1995. at a joint meeting of senior officials, U. S. Publications in 1995/1996 Departments of Education and Labor. Washington, DC, 1993. Brown, David E. “State Programs in School- ______. Key Principles in Building a to-Work System Development,” in State Youth Development System. Paper pre- Line, July 1995. sented at the National Youth Employment Burtless, G.T. “The Employment Prospects Coalition Policy Forum. New York, NY, of Welfare Recipients,” in The Work Al- 1994. ternative: Welfare Reform and the Reali- King, C. T. and R. McPherson. An Analysis ties of the Job Market edited by D. of the Austin Metropolitan Area Labor Nightingale and R. Haveman. Washing- Market. Austin, TX: Austin Area Re- ton, D.C.: Urban Institute Press, 1995. search Organization, Inc., 1990.

Evaluation Forum n Issue 12 n Summer 1997 37 Interview Interview

Editor’s Introduction This interview focuses on this new merous articles that have captured the context for youth, particularly the envi- serious attention of both scholars and One of the major tasks of societies ronment in which poor youth in U.S. cit- public policymakers. He has proposed is the socialization of its youth--that is, ies are growing to adulthood in the a research-based theory of the social the teaching of youth to play responsi- 1990s. The discussion is informed transformations which have occurred in ble, meaningful, satisfying adult roles, uniquely by a comparative analysis the inner city, and has recommended a however those roles are defined by a across several European societies, comprehensive policy agenda ad- particular society. Western civilization based on William Julius Wilson’s chap- dressing broad social problems which and the U.S. society in the 1990s are ter in his new edited book, Poverty, Ine- disproportionately affect the inner city experiencing new problems in the so- quality and the Future of Social Policy, poor. cialization process, such as a high lev- published in 1995 by the Russell Sage His book titled The Declining Signifi- el of drug abuse and crime, and the Foundation. cance of Race: Blacks and Changing adoption of alternative values and William Julius Wilson is currently American Institutions received the roles whose definitions vary from the the Malcolm Wiener Professor of Social 1980 Sydney S. Spivak Award. His central tendency shaping notions of re- Policy at the John F. Kennedy School of 1988 book, The Truly Disadvantaged: sponsibility. Increasing employment Government at Harvard University. He The Inner City, the Underclass, and challenges, growing wealth and in- was formerly Lucy Flower University Public Policy received the C. Wright come inequality, and rising poverty in Professor of Sociology and Public Poli- Mills Award and was selected by the Western societies have limited the life cy at the University of Chicago. Wilson editors of the New York Times Book Re- chances of youth, complicating critical is an elected member of the National view as one of the sixteen best books socialization tasks. Our explanation of Academy of Sciences and the Ameri- published in 1987. His latest book is “adulthood” is less clear. While this can Philosophical Society. He is also a When Work Disappears: The World of new flexibility gives greater discretion Fellow of the American Academy of the New Urban Poor, published in 1996 to youth in making decisions about Arts and Sciences and the American by Alfred A. Knopf. themselves and others, and about their Association for the Advancement of He is the recipient of the 1987 Mac- own life directions, it is emerging at a Science. high price in terms of personal and so- Arthur Prize and the 1988 Godkin Lec- cietal safety. Directing most of his work to the tureship at Harvard, and is past presi- study of race, class and poverty, he has dent of the American Sociological written several major books and nu- Association.

38 Evaluation Forum n Issue 12 n Summer 1997 Interview

Question: What led you to produce a In Germany, foreign workers and mainstream labor market institutions new book on poverty and inequality? their families were much less likely and informal job networks. This than native Germans to be “officially spawns a cycle of weak labor force at- Response: We found the nature of poor.” Subsequently, however, and par- tachment, increasing social exclusion, the 1980s’ American social policy de- ticularly in the 1990s, this changed dra- and rising group tensions. Trends in a bates concerning poverty frustrating matically. During 1991, almost 15% of number of European countries suggest and disturbing. They seemed disjoint- all foreigners utilized public assistance the beginning of urban social polariza- ed, some positions emphasizing the im- at some point; only 5% of the German tion, leading to a concentration of social pact of economic change, others giving population needed to depend upon it. problems in certain areas of cities. Eu- primary attention to moral decline. Also, foreigners have been dispropor- ropean research shows that as poverty Meanwhile, structural factors such as tionately affected by the cuts in social becomes more concentrated it involves income and wealth inequality, rising benefits — that is, in means-tested pov- multiple kinds of deprivation, which worker displacement and joblessness, erty programs — in which foreign work- then translates to both material decay and a growing concentration of poverty ers and their families have tended now and reduced life quality. This cycle and in urban areas were having significant to be overrepresented. In the United these trends have important conse- effects on life chances. We felt that a Kingdom, the Thatcher government in- quences for the socialization of the comparative analysis across industrial- troduced major cost-cutting reforms in next generation. ized countries would focus attention on the context of 8 million people (one in these institutional factors and their im- 7) of the total British population living Question: What are some of the social plications for poverty and inequality. policy differences between Europe and in households dependent on a national the U.S. that have mediated against The Rockefeller Foundation provided means-tested and last-resort income sharp socio-cultural divisions within the initial funding for this policy study, support program. This is a significant European cities, and the concentration and several other foundations subse- increase over 4 million in 1973 and 1.2 of poverty in particular parts of these quently contributed to the project, in- million in 1950. cities, with all the attendant social cluding the Russell Sage Foundation. Worsening economic conditions in problems? This permitted important meetings of Europe have led to an urban housing the project’s advisory committee and crisis and insecurities in the labor mar- Response: As the political scientist the international group of experts who Margaret Weir has pointed out, in ket, as well as a perception that the co-authored the book. We hoped that American cities local land use and hous- growth of the minority and immigrant the book would change the character ing policies, and federal and state urban populations has been the major cause. of the policy debate, away from a pre- policies, have made it easier to separate The recessions have also exposed the occupation with individual attitudes groups by income, race and ethnicity. vulnerability of ethnic communities to and behavior and toward an analysis of U.S. cities have much greater autonomy economic stagnation, industrial restruc- the economic context influencing the and control over population movements turing and the decline of traditional nature of poverty in contemporary soci- and urban development than European manufacturing. Related to these devel- ety. urban areas — such as through educa- opments, immigrant unemployment tional policy, zoning ordinances, and rates have been reaching 25-50% in cit- Question: The economic and labor other strategies manifestly designed to market influences that have condi- ies experiencing high levels of employ- ensure community life quality. Federal tioned poverty increasingly in Western ment for the general population. Many policies have supported such autono- societies have been of particular inter- of the immigrants most acutely affected my. est to you. What insights does this new by recession have been second and look at European poverty provide in third generation minority youth with For example, in the U.S. federal fi- this respect? educational deficits, who are having nancing for freeways and subsidies for difficulty finding employment and are home ownership contributed to the de- Response: Since the late 1970s, Euro- isolated and alienated from the societ- velopment of urban suburbs and new pean governments have faced the re- ies of the older generations. But these political jurisdictions. Federal acquies- turn of mass unemployment and a conditions for immigrant youth primari- cence to community organizations’ re- growing fragmentation of their labor ly reflect deteriorating economic condi- sistance to public housing in middle markets. Increasing numbers of indige- tions and declining organizational re- class neighborhoods contributed as nous workers and immigrants have fall- sources in the poorer working-class well, as did federal toleration of exten- en into the secondary labor market — communities. sive segregation of blacks and other mi- characterized by low skills, poor norities in urban housing markets. This working conditions, weak union protec- However, Europe and the U.S. are in led to large-scale segregated housing tion, and a high risk of unemployment. marked contrast in terms of urban ine- projects that became disadvantaged mi- European unemployment rates are now quality. No European city has experi- nority areas concentrated in overcrowd- close to ten percent. At the same time enced the level of concentrated poverty, ed, deteriorating inner-city ghettos. there has been a growth in long-term and racial and ethnic segregation, that And since the 1980s, the federal gov- unemployment, forming a substantial characterizes American metropolitan ar- ernment has withdrawn support for ba- subgroup of individuals with loose at- eas. No European urban area incorpo- sic urban programs, which has in- tachment to the labor force (including rates areas as physically, socially and creased the concentration of poverty in many young people). In fact, in 1993 culturally isolated, deteriorated or vio- these urban areas. This sharp drop in half of all unemployment in Western lence-prone as American inner city ghet- support for such programs in turn con- Europe was long-term, compared with tos. However, many European inner- tributed to the declining political influ- 10-20% in the rest of the industrialized city communities and suburban public ence of cities and the rising influence world. This subgroup included many housing areas have been cut off from of suburban political power bases. young people.

Evaluation Forum n Issue 12 n Summer 1997 39 Interview

In Britain, a strong central govern- the rate of most European countries and and cultural diversity more critical to ment has exerted significant control almost three times the West German consider. Now the universalistic Euro- over population movements and urban rate. The U.S., in fact, had the highest pean welfare systems are undergoing development. Local authorities have lit- national poverty rate of all thirteen significant change, with an increasing tle autonomy. Since the condition and countries included in the database of marginalization of certain groups and quality of urban educational institutions the 1991 Luxembourg Income Study. The the potential emergence of a racialized are determined less by local govern- severity of poverty in the U.S., and the economic underclass. Rising unemploy- ment budgets than by national financ- marked difference between whites and ment and economic fragmentation in ing, there is less competition between racial minorities in the concentration of European countries pose a test of the cities for scarce resources. Furthermore, poverty, were significant findings. ability of these welfare states to the construction of urban housing in counter marginalizing tendencies. The still-dramatic differences be- Britain has involved mainly public hous- tween U.S. and European poverty are For example, an interdisciplinary ing, which made the mixing of people likely associated with varying antipover- study of poverty and marginalization in with different incomes inevitable. And ty policies. European policies have been West Germany revealed the effects of public housing was less concentrated directed more broadly toward the work- economic dislocation on citizens’ social in particular areas--it was not deliber- ing class. The prevailing view in Europe rights. Historically, the “productive ately segregated. For example, by the following World War II has been that core” of German society, in terms of late 1980s, while 43% of public hous- the best policies for assisting the disad- stable employment, was in control of ing was in poorer inner London, 29% vantaged are policies applied through the social insurance system. Those was in central London, and 23% was programs benefiting the population as a more vulnerable to unemployment were outside London city limits. whole — that is, universalistic policies, in large part excluded from many facets In France, the disadvantaged tend to rather than policies leading to specifi- of this system. Benefits for this growing be concentrated in city suburbs, with cally-targeted, means-tested programs. vulnerable group were accessible main- in-city housing viewed as the most de- These universalistic policies have sup- ly through means-tested public assis- sirable. Although poor ethnic and racial ported macroeconomic strategies that tance. Although 25% of workers were minorities tend to be concentrated in address general concerns about labor eligible for these programs in the mid- urban high-rise public housing, French force and labor market issues. And 1980s, more than half of these workers land use and housing policies have con- these policies have tended to be linked had not applied due to the stigma at- trolled population movements more more closely with social security, health tached to non-universalistic welfare than in the U.S., and permitted more and housing policies reinforcing the strategies. This new division of the Ger- income and ethnic mixing than in the working and middle classes. man welfare state into two types of U.S. or Britain. workers with different welfare benefits Although the influence of trade has raised the possibility of ghettoizing This is changing, however. Recent unionism in assisting vulnerable groups marginal workers, particularly a grow- population movements and government in the labor market has eroded both in ing number of East Germans and an in- action in Britain and France have in- the U.S. and Europe, this development creasingly large foreign subpopulation. creased the concentration of poverty in the U.S. reflected greater pessimism During 1991, nearly 15% of all foreign- and racial/ethnic segregation, much as about alleviating poverty and mounting ers were receiving public assistance at such policies separated populations in effective welfare programs. This pessi- some point; less than 5% of the Ger- the U.S. It is unlikely that Britain and mism flowed from the use of a limited man population was doing so. In the France will experience the degree of in- range of antipoverty strategies due to context of the higher use of public as- come and racial/ethnic separation and American resistance to universalistic so- sistance by foreigners and their fami- segregation that characterizes U.S. met- cial service and labor market interven- lies, the Solidarity Pact of 1993 involved ropolitan areas, however. Central gov- tions. Largely ignored in the U.S. were cuts in means-tested programs, which ernments are likely to maintain control the effects of basic economic and tech- further isolated certain groups from over the development of new political nological changes that were affecting mainstream citizen rights and benefits. jurisdictions, reducing incentives and the work opportunities, experiences opportunities for such separation. Also, and prospects of the poor. Absent from In the United Kingdom, the Thatch- suburbanization in Europe has not tra- anti-poverty policy was the European er government consciously exacerbated ditionally involved an abandonment of concept of comprehensive programs de- the division of the welfare state by en- cities as desirable residential areas. signed to promote the social rights of couraging tax-supported private and Rather they are treated as national re- citizenship. company-based welfare among the sources to be protected. middle class and more affluent work- Europe’s postwar commitment to ers, along with major cost cutting in so- universal programs and to social inclu- Question: How have European anti- cial security and housing. This shifted poverty policies differed from those in sion, however, was based on assump- the welfare state further toward more the U.S.? tions of steady growth and full employ- targeted welfare programs. The same ment, on improvements in employment It is clear that poverty and trend has occurred in France and the Response: and labor market realities, and on no- inequality have increased in Europe as Netherlands. Nevertheless, as I pointed tions of collective solidarity and cultural well as in the U.S., related to econom- out earlier, although there is a trend to- homogeneity, which de-emphasized ic, social and political forces. However, ward the more limited social provision historic divisions between religious and by the second half of the 1980s, the of the U.S. system, the European com- ethnic groups. These assumptions were American poverty rate among the non- mitment to universal social services and weakened by economic change and the elderly population was more than twice broad family policies remains a firm in- absorption of a larger number of immi- stitutional feature. grants, making issues of race, ethnicity

40 Evaluation Forum n Issue 12 n Summer 1997 Interview

Question: What do you mean in the Another critical factor in under- division. Although government benefits book by the emergence of “new poverty” standing the new poverty is the mas- were improved, little was proposed for in the U.S. and Europe? sive breakdown of social institutions in integrating clients into the social and poverty neighborhoods. An important economic mainstream. Since poverty Response: The period since the late aspect is the absence of nonpoor fami- was not viewed as a national economic 1970s marks an important juncture in lies in ghetto areas which could serve problem, antipoverty programs focused addressing poverty and inequality is- as conventional models in socializing on efforts to change the behavior of the sues in the western world. Economic poor youth. The absence of diverse so- poor rather than their economic oppor- insecurity has been accompanied by cial resources and crucial community tunities. family disintegration and altered forms networks has weakened youth’s attach- of public assistance, reducing the life The targeted programs that became ment to the legitimate labor market. chances of an increasingly vulnerable the traditional American response to minority population. A variety of eco- Question: What do you see as the ma- poverty created a policy dualism that nomic, social and political forces have jor characteristics of American anti- made it easier for the middle and upper been producing powerful new configu- poverty policies and how effective do classes to criticize the welfare system rations of inequality which we have re- you feel they have been? and its clients. The welfare backlash of ferred to as “the new poverty.” These the 1980s and 1990s is also a product of configurations reflect changes in the Response: In 1975, 30% of all Ameri- antipoverty strategies that fostered in- can poor had incomes below 50% of size and composition of economically tergroup misunderstanding and mis- the official poverty level; in 1988, 40% marginal groups, the crystallization of trust, as well as the isolation of the had such incomes. The increase was racial cleavages among them, a down- poor. As one expert put it, we are in a much greater for blacks — in 1975, ward turn in their life chances, and an “retreat from caring.” To many middle nearly half had incomes at that level. increase in their social and political iso- Americans, the poor had become “the These effects are strongly tied to post- lation. welfare class” — that is, the bottom war American antipoverty policies that level of the stratification system defined As poverty rates have risen in most have led to relatively autonomous ar- in terms of their behavioral and cultural western societies, the rates among rangements for the poor in isolation deficiencies. This “class” was consid- young people and prime-age workforces from broader social concerns about eco- ered to pose a growing threat to public have grown disproportionately, particu- nomic, educational and employment is- norms and safety. The emphasis on the larly among single parents with chil- sues. In fact, we have focused on de- personal characteristics of individuals, dren. In the U.S., the movement away veloping programs rather than on particularly their work ethic, ignored from manufacturing toward a service developing national policy — particular- the large-scale economic changes that economy, the increase in the proportion ly decentralized and non-integrated pro- were contributing to increasing poverty. of low-wage and part-time employ- grams and demonstration projects. At the same time, research was strong- ment, widespread worker dislocation, Historical analyses of the American ly suggesting that there was no conclu- and rising long-term unemployment in approach to poverty consider this ap- sive evidence for prevailing beliefs that certain segments of the labor force proach a legacy of the New Deal. The welfare participation discourages work, have been particularly traumatic for the nationwide social security system was divides families or increases the child- urban black poor because of their con- an extension of the rights of a predomi- bearing rates of unmarried women. centration in deteriorating inner-city nant group of citizens, the “deserving” ghettos. Consequently poverty is affect- Two themes have now emerged: working class. They had the support of ing the opportunities and future of the 1) welfare receipt should involve cer- the middle class and a strong federal poor more dramatically than in the re- tain responsibilities on the part of the bureaucracy. However, the New Deal cent past. Some experts see a “femini- recipient, and 2) able-bodied adults re- did little for those with low skills and zation” and “racialization” of poverty ceiving welfare should be required to social status, including blacks moving which revolves around labor market develop educational and work skills, to to northern cities from the rural south. and labor force relationships. search for jobs, and to accept the jobs Attempts to develop a national full em- offered them. Both themes are based In this new environment, the “moral ployment policy and health care pro- on pre-existing assumptions about the worth” of the poor is scrutinized more gram in the 1930s and 1940s, which moral fabric of individuals, independent closely in means-tested programs — could have assisted the disadvantaged, of the social, economic, cultural and that is, their work ethic, family relation- were not politically viable. Neither political structure of society. ships and integrity. This is changing the were U.S. laws, union power or cultur- poor’s attitudes toward government al traditions supportive of the disadvan- While economic restructuring and and the welfare system, with these in- taged, as they were in Europe. Assis- downsizing have had adverse effects stitutions being viewed as hostile adver- tance to this group was confined to on lower-income groups in all advanced saries. Rather than the welfare system poorly coordinated second-tier programs industrial nations, the effects of jobless- having evolved into a “dependency cul- under the “welfare” system. This re- ness on the American poor have been ture,” research indicates that it is be- stricted the scope of anti-poverty strate- far more severe than elsewhere be- coming a “captivity culture” in which gies and reinforced the American divi- cause of the underdevelopment of the welfare claimants may be trapped into sion between the “undeserving” and welfare system and the weak institu- dependency against their will and poli- “deserving” poor, the former acquiring tional structure ensuring the general so- cies designed to promote self-sufficien- the status of a class apart. cial rights of citizens. The tradition of cy may reinforce dependency as a man- noncomprehensive programs in the U.S. In many ways, the Great Society ageable phenomenon. is particularly problematic for poor in- programs of the 1960s continued this ner city blacks who have been handi-

Evaluation Forum n Issue 12 n Summer 1997 41 Interview

capped by intergenerational discrimina- Additional books and articles on tion and find themselves isolated in Editor’s Note comparative poverty issues: ghetto neighborhoods. The first issue of Evaluation Forum Brown, J. (editor). Anti-Poverty Policy in the Question: What conclusions do you focusing on poverty featured an inter- European Community. London,UK: Policy come to in the book about effective anti- view with William Julius Wilson based Studies Institute, 1984. poverty policies and strategies? on his classic, The Truly Disadvan- Glazer, N. and K. Young (editors) Ethnic Plu- taged. For those interested in a reprint ralism and Public Policy. Lexington, MA: Response: Any program proposing to improve the life chances of the poor, of that interview, please contact the ed- D.C. Heath, 1983. including the ghetto poor, needs to be itor of Evaluation Forum. Katz, M. The Undeserving Poor: From the based on concerns about both the poor Readers may also be interested in War on Poverty to the War on Welfare. and the working classes. Unemploy- the following books and articles au- New York, NY: Pantheon, 1989. ment, job security, declining wages, ris- thored, co-authored or co-edited by Klass, G. “Explaining America and the Wel- ing medical and housing costs, the Wilson: fare State: An Alternative Theory,” in Brit- need for child care, the sharp decline ish Journal of Political Science 15, 1983. in the quality of public education, 1987: The Truly Disadvantaged: The Inner City, The Underclass, and Public Policy. Korpi, W. “ Social Policy and Distributional crime and drug abuse — these broader Chicago, IL: University of Chicago Conflict in the Capitalist Democracies,” problems must be addressed. Reliable Press. in West European Politics 3, 1980. national surveys indicate wide support 1991: “Studying Inner-City Social Disloca- OECD: The Future of Social Protection: The for general strategies to resolve employ- tions: The Challenge of Public Agenda General Debate. Paris, France: OECD, ment problems, such as educational Research,” in American Sociological Re- 1988. and skill training, the provision of day view 56. Rainwater, L. Poverty in American Eyes. care, the reform of the health care sys- 1995: Poverty, Inequality, and the Future of Luxembourg Income Study, 1991. tem, and crime and drug addiction pre- Social Policy: Western States in the New vention. This suggests possibilities for Standing, G. Unemployment and Labour World Order . (W.J. Wilson with K. Mc- Market Flexibility: The United Kingdom. new demographic and political align- Fate and R. Lawson, editors.) New York, International Labour Office. Geneva, ments and coalitions that can place NY: Russell Sage Foundation. Switzerland, 1986. greater emphasis on the integration of 1996: When Work Disappears: The World of programs promoting social and econom- Weir, M. “Race and Urban Poverty: Com- the New Urban Poor. New York, Alfred A. paring Europe and America.” Occasional ic progress for all groups in society, not Knopf. paper. Center for American Political just disadvantaged segments. Studies. Cambridge, MA: Harvard Uni- In Europe, income transfers, family versity Press, 1993. policies and health benefit systems re- Weir, M., S. Orloff, and T. Skocpol (editors) ceive wide popular support, and in- The Politics of Social Policy in the U.S. clude an array of related program strat- Princeton, NJ: Princeton University egies. In the Swedish reform package, Press, 1988. labor market programs still play a key role. In Germany, the tradition of active labor market programs and close linkag- es across government, educational insti- tutions and industries have continued, but with significant cost cutting. When Germany’s vocational vs. higher educa- tion dual system was failing to move many young people into permanent mainstream employment, a well-direct- ed trade union and employer effort has begun to restore its health. In France, the United Kingdom and the Nether- lands, there has been a movement to improve job skills and opportunities even though there has been a negative reaction to the welfare state in terms of its perceived role in increasing depen- dence. Unfortunately many of the reforms that focus on more universalistic strate- gies, such as labor market programs, have been poorly designed and their delivery suffers from unprecedented de- centralization. These programs can be effective only as part of broad national programs promoting economic recovery and adjustment to structural change.

42 Evaluation Forum n Issue 12 n Summer 1997 Evaluation Issues and Activities n International Perspectives on Youth Issues Evaluation Issues and Activities

International Perspectives on Youth Issues nn Editor’s Introduction people past their mid-20s. In 1992, An Overview of roughly one out of every thirteen juve- In March, 1996, the U.S. General Ac- niles reported having been a victim of Research on counting Office (GAO) published a re- a violent crime. Meanwhile, the U.S. port on at-risk and delinquent youth. It has the highest teen alcohol-related International Youth was revealing, if not shocking. Accord- and drug abuse-related rate of any in- Issues ing to the GAO, approximately 20% of dustrialized nation. children in the U.S. live in poverty; for The federal government currently minority children the percentage is operates 131 programs intended to Two books on international youth is- twice as high. Poor teens are more than benefit youth, including at-risk and de- sues provide interesting ideas to consid- three times as likely as other adoles- linquent youth. These programs are ad- er in terms of youth development and cents to drop out of school. While the ministered by 16 different federal de- programming. One was published by high school dropout rate has been fall- partments and agencies. Many of these the Organization for Economic Coopera- ing, in 1993 it remained over 10% for all programs have similar goals, and pro- tion and Development (OECD), the oth- races combined, and over 27% for His- vide multiple services to multiple target er by the United Nations. panic Americans. groups, although the majority target all The unemployment rate for youth is youth.The GAO indicates that the feder- The United Nations much higher than for adults. From 1991 al government spent over $4 billion on to 1994, the average unemployment programs for at-risk and delinquent The International Year of Youth was rate among males 16 to 24 years old youth in 1995. celebrated in 1985. This nominal focus on the world’s youth prompted the was 14.3%, compared with 5.7% for The Congressional request to the those aged 25 and older. During the United Nations to initiate the United GAO was simply to list and describe all Nations University’s Project on Youth. same period, the average unemploy- these federal programs without offering ment rate among females 16 to 24 was The project produced a book reviewing recommendations. The clear intent of expert opinion and research on youth 12.3%, compared with 5.3% for those the House Committee on Governmen- 25 and older. issues titled Perspectives on Contempo- tal Affairs in making the request, how- rary Youth. Edited in 1988 by J. Kuczyn- In 1992, child protective service ever, was to determine subsequently ski, S. Eisenstadt, B. Ly and L. Sarkar, it agencies received approximately 1.9 what programs could be eliminated or featured a chapter by E. Allardt which million reports of child abuse and ne- consolidated. It is interesting, in this may be of special interest to readers, glect. In 41% of these reports, investi- context, that the GAO chose to provide “The Current Context of Youth in Econ- gations confirmed maltreatment. Re- extremely troubling data on the status omy, Politics and Societal Develop- search has suggested that such abuse of youth in America in the mid-1990s. ment.” and neglect increases the likelihood of The section that follows provides juvenile arrests by 53%. Between 1988 some international perspectives on and 1992, the juvenile arrest rate for vi- The Transition from Youth to youth issues, which help place the Adulthood olent crime increased 47% — more American statistics within a broader Allardt maintains that although the than twice the increase for people 18 context. It also addresses strategic length and nature of the transition peri- years of age and older. The number of planning and methodological issues in od from childhood to adulthood has violent crimes committed against youth analyzing the “performance measure- varied in different societies, it is today aged 12 to 17 increased 25%. In 1991, ment movement” and its relationship to associated with more serious prob- youth aged 12 to 23 were more likely to evaluation. be victims of a violent crime than were lems — “being young and becoming an adult seem to be more difficult than during many other periods in history.” This transition is highly dependent, he says, on economic and social condi- tions. He reminds us that there are sig- nificant differences in opportunity for youth in rich vs. poor countries. For ex- ample, although unemployment is a problem everywhere, in developing countries underemployment or “ill-em- ployment” is the dominant characteris-

Evaluation Forum n Issue 12 n Summer 1997 43 Evaluation Issues and Activities n International Perspectives on Youth Issues

tic. Currently, even in the industrialized dle class in the context of the decline of labor market change in terms of educa- countries, unemployment has reached its socioeconomic position. tion and skills. Therefore the bound- unexpected high levels, particularly aries between adults and youth have among youth. The Youth Labor Market grown vaguer. Technical improvements in agricul- Allardt sees the nature and degree of Although Allardt takes the position ture are leading to greater rural unem- youth unemployment as conditioned that young workers are more sensitive ployment and rising migration to urban strongly by demographic factors. Many to working conditions than their elders centers. Employment has shifted from developing countries have a larger 20-24 — both physical safety and alienation — agriculture to manufacturing and now age group than can be absorbed by their he also points out that youth are in the to services. Approximately two-thirds of labor markets. In advanced industrial- forefront of technological change, which the U.S. labor force is currently in the ized countries, the “baby boom” gener- translates to new job opportunities. service sector; approximately 60% is in ation entered the labor force in the mid- What the trade-off will be between such services in Western Europe. Electronic 1970s when labor demand was weak. change and unemployment is yet to be data processing is used increasingly in Evidence from a number of countries determined. all sectors of economies. reveals a growing trend toward long- These major changes, Allardt says, term unemployment in the 20-24 age The Organization for Economic have had a number of consequences group.This shift in the age structure of Cooperation and Development for the transition of youth to adulthood. unemployment, Allard believes, is likely to continue. The status of youth in European Changes in the structure of employ- Union countries and other industrialized ment which have restricted opportuni- In fact, research indicates that youth nations has been a longstanding inter- ties for social mobility, have affected unemployment may be more troubling est of the OECD. In 1989, representa- youth’s view of their career opportuni- than the statistics suggest. Youth have a tives of seventeen OECD member coun- ties and life chances. There has been a tendency to leave the labor market tries met in Paris to develop a project substantial expansion of government, when facing difficulties. In most coun- on at risk youth under the sponsorship with the public sector assuming many tries there is a sizable minority of young of OECD’s Centre for Educational Re- of the social functions formerly per- people who experience serious prob- search and Innovation (CERI). This formed by family and community. Con- lems during the transition from youth to meeting was followed by two interna- sequently current attempts to reduce adulthood and consequently live a job- tional seminars supported by the re- the size and change the nature of pub- lessness or casual labor employment ex- search arm of the U.S. Department of lic social safety nets will likely have a istence. Those with low educational at- Education and several major founda- negative effect on the employment pos- tainment and physical and emotional tions. sibilities of young people. handicaps, and ethnic minorities and The project focused on three educa- Allardt proposes that economic teenage parents, are overrepresented in this group. tional/training periods: preschool, change, change in the structure of em- school, and the transition to work. Its ployment, and an increased role for Allardt points out that maintaining purpose was to identify key issues in government has influenced youth atti- young people in educational institutions studying youth problems, and to make tudes. Some of the elements of youth’s has been one of society’s main devices recommendations for national and inter- new orientation are these: for coping with the problem of unem- national youth policy. In 1995, CERI ployment. The amount of formal educa- produced a book based on the study n An emphasis on inner-direction, compared with sensitivity to manip- tion now required by postindustrial la- commissioned by OECD, Our Children ulation by external demands. bor markets assigns additional tasks to At Risk. The terms “at risk” referred to these institutions — intellectual, aes- youth from disadvantaged backgrounds n A stress on individual human rights. thetic, ethical, psychological, and other who are likely to fail to meet school tasks. At the same time, the motivation standards, often drop out of school, n A focus on protecting the fate of the individual -- i.e. individual safety, to learn is becoming increasingly depen- and frequently fail to become integrat- physical, psychological and social. dent, he says, on the characteristics of ed into a normally accepted pattern of education (its quality) rather than on its social responsibility, particularly with n An interest in causes, without con- prospective rewards. regard to work and family life. sideration of material interests, well- being or rights. A recent study by OECD drew the conclusion that youth’s most powerful School-to-Work Cooperation These new attitudes, particularly demand was for full integration into the and Linkages among youth in industrialized coun- adult world, and that this demand has The European Union has been trou- tries, have been played out in anti-bu- been moving down the age scale. In- bled in the 1990s by the proportion of reaucratic, environmental, anti-nuclear, creasingly younger age groups have at risk youth in member countries. Up feminist and peace movements, as well now ceased to define themselves as to thirty percent of school age children as in religious and ethnic revivals, ac- children, while the upper limits of sec- are estimated to be “at risk.” Even in cording to the author. These attitudes, ondary and postsecondary education countries where unemployment has he says, appear to be reactions against have moved upwards — consequently fallen there has been a rise in unem- structural and institutional trends: the the demand for adult status and the real ployment for ethnic minorities and increasing importance of service and possibility of acquiring it are farther youth. Any gains made in the educa- goods; the penetration of government apart. Even adults can no longer feel tional system are sometimes sacrificed into most spheres of life; and the in- fully prepared for rapid economic and in the transition period from school to creased size of the well-educated mid- work.

44 Evaluation Forum n Issue 12 n Summer 1997 Evaluation Issues and Activities n International Perspectives on Youth Issues

European countries have traditionally at risk youth adequately, according to nn relied on strong relationships between the OECD report. Nevertheless, some Youth Training schools, industries and businesses in countries have developed useful mod- An interesting cross-national study terms of developing relevant skills. For els. For example, Portugal has devel- supported by the U.S. Department of example, in the United Kingdom, pro- oped an interministerial program that Labor has been conducted by The grams have been developed to give unifies the national approach at central, Rand Corporation to study the self-re- school curricula a more job-oriented di- local and school levels, and is coordi- ported formal job training and labor mension along the lines of the Boston nated by a central interministerial coun- market outcomes of male youth in the Compact. In France, the educational sys- cil. This approach gives priority to basic United States, Britain and Australia. tem emphasizes the consistent guidance education and to areas of low educa- The report on the study was published and monitoring of youth, work study tional performance, and involves a link- in 1991. Three national surveys provid- programs, and an individualization of age between the educational system and ed the data: the U.S. National Longitu- educational plans. In Sweden, students health and nutrition programs, recre- dinal Survey of Young Men (NLS), the are required to have a series of weeks in ation programs, family support pro- British National Child Development a work experience setting. In Japan, grams, and transportation systems. Study (NCD) and the Australian Longi- special schools and training colleges tudinal Survey of Youth (ALS). Informa- linked with businesses have been devel- Some Conclusions tion was available on educational at- oped to offer flexible opening times for tainment, participation in formal students prior to and following work, Extending the capabilities of school training programs, wages, job charac- and alternatives to drop-outs. Often systems in responding to the needs of teristics and employment. In addition, community initiatives have supplement- disadvantaged students has led many panel data on other relevant variables ed the above linkages. In the United countries to review the balance of con- was analyzed. The study was guided Kingdom, special programs for inner trol between central and local de- by propositions about the relationship city youth have been developed. In Aus- mands. Many countries see the mainte- between determinants of training and tralia, community centers have been es- nance of quality as a significant the effects of training on wages and tablished. challenge. Some have resolved this is- sue by emphasizing local control while the likelihood of employment. In the Netherlands, new policies link requiring some form of accountability to a number of government ministries the central government. Research Questions around these purposes: An array of intervention strategies Several interesting policy questions n To coordinate national policies on have been developed in OECD coun- directed the study: youth education. tries, including service coordination, in- n Do youth compensate for a low level creasing continuity across major transi- n To collaborate with trade unions and of education through job training, or tion periods, using flexible approaches, employers around more effective uti- do those lacking education lose ac- giving greater autonomy to schools, lization of existing programs and in- cess to future occupational training? frastructures. changing school organization, develop- ing new curricula, assigning more im- n What role does technology play in n To combine budgets. portance to youth’s personal and social determining workplace educational In Australia, cooperation across needs, and giving more freedom to and skill needs? teachers to develop more effective schools, universities, research organiza- n Do high union wages and work rules teaching strategies and better linkages tions, trade unions and businesses is reduce employers’ incentive to pro- with employers. These interventions of- part of a general strategy to broaden vide training? school-to-work options. In Germany, co- ten involve systemic changes in the ordination policies involve welfare school-to-work sequence, and its insti- n How do training patterns differ in agencies, businesses, schools, and em- tutions and organizations, and the use the three countries and what policy ployment and training programs. In Por- of educational and training resources. lessons are suggested by commonali- tugal, there is extensive development of ties and differences? multi-agency community programs to encourage the social and economic inte- Editor’s Note Study Results gration of minorities. A statistical analysis of the economic Please see Part II of the Resources However, a lack of cooperation and determinants of training in the three Section for international references on countries indicated that the probability coordination across multiple actors con- youth issues. tinues to be a major barrier to serving of receiving most kinds of formal train- ing rose with the level of schooling re- ceived, with the possible exception of Australia where education and training appeared to complement rather than substitute for one another. One implica- tion of these results was that youth with low schooling also confront limited training opportunities and consequently a slower growth in wages. The probabil- ity of obtaining training, however, peaked at different schooling levels for

Evaluation Forum n Issue 12 n Summer 1997 45 Evaluation Issues and Activities n International Perspectives on Youth Issues

each source of training, and training bringing skills from previous jobs to the U.S., training courses taken in institu- probabilities varied by apprenticeship new one. tions following the completion of status in Britain and Australia. schooling had no apparent effect on Controlling for other factors, white- wages. The results were similar for the There was a dramatic pattern of ef- nonwhite status did not appear to affect other two countries. However, the size fects on training by the source of train- access to most formal training. High un- of wage returns associated with the re- ing. The likelihood of company training employment increased the likelihood of ceipt of training revealed large differ- was greater in high technology jobs, es- being trained in a business or technical ences across the three countries. The pecially for those with more education. institution in the U.S., but reduced wage effects of formal training in the Rapid technological change clearly pe- training for nonapprentices in Britain. U.S. was about twice those in Britain nalized early school leavers. High tech- Respondents in large firms were more and Australia. This could explain why nology jobs were correlated also with a likely to receive formal training from all employers and young workers estab- lower likelihood of off-the-job training. training sources. Private sector employ- lished in the workforce in these coun- These results suggested that in a grow- ment in both Britain and Australia, tries have less incentive to provide and ing and technologically progressive en- however, was usually associated with a receive training, since the returns for vironment, employers rely more on lower likelihood of receiving training. schooling and other forms of invest- company training for skill needs than The research showed that in the U.S. ment may be as high as those for for- on schools and other outside sources of wages rose with both work experience mal training. training. and job tenure, although at a slower In all three nations, the likelihood of For American male youth, the proba- rate at the higher work experience lev- experiencing a spell of unemployment bility of receiving training from all els. Results for Britain suggested that in a given year was lower for those sources rose with years of work experi- each year of work experience increased with higher schooling, more general ence, holding job tenure constant. For wages by about the same amount as in work experience and longer job tenure. British youth, only company-based the U.S. In Australia, wages grew rapid- Employment in large firms was associ- training appeared to rise with work ex- ly with years of work experience but the ated with greater stability. Membership perience, and then to decline. And the effect lessened quickly. Job tenure had in unions was related to a higher proba- likelihood of training from most other no statistically significant effect on bility of unemployment. Higher rates of outside sources decreased with work ex- weekly wages in Britain or Australia. technological change in the U.S. and perience. Much of the cross-national difference in Britain were associated with a lower wage profiles seemed to be due to the Employers appeared to use different likelihood of unemployment. In the larger wage-tenure effects in the U.S. , training strategies in the three coun- U.S. this was true mainly for selected which may mean there is considerably tries. In the U.S., after a beginning schooling groups, suggesting that there more informal on-the-job training in training period in the first year on a was less technology-induced youth un- the U.S. than in the other two countries. job, training from schools and busi- employment than expected. ness/technical institutes decreased Training appeared to produce higher Training reduced the likelihood of with time in the job, while company- marginal wage returns to labor market unemployment in all three countries. based training and training from other entrants with low schooling, possibly Each training episode appeared to low- sources rose with tenure. This pattern because training tended to substitute er the probability of future unemploy- suggested that company training was for school at that level of educational ment, particularly if it was job-related, substituted for broad-based general attainment. Also, a large part of the with a slight decline in this effect over skills acquisition supplied elsewhere. training provided seemed to be quite time. British and Australian employers tended general. Transferable skills appeared to to rely on outside training sources, with be related more to time in the labor company training falling with job ten- market than to job tenure. The wage Conclusions ure, while off-the-job training rose ini- returns for a high level of schooling In all three countries, the probability tially and then declined. were clearly higher in high-technology of receiving training from all sources in- jobs. Technological change actually re- Unexpectedly, the study showed creased with the level of educational duced the wages of the least educated. that in all three countries union mem- attainment. Union involvement in- The research suggested that better-edu- bership or coverage was associated creased access to training. Although cated workers, in all three countries, with a greater probability of receiving American male youth received much were better able to respond to techno- formal training from most sources. This less training at entry to the workforce, logical change, and were more produc- suggested a more extensive training role they accumulated training at a much tive and more highly rewarded. for unions than had been anticipated. more rapid pace with time in the labor In each country, wages for nonwhite force. In fact, their likelihood of receiv- For all countries, the probability of male youth were 10% lower than for ing additional company training re- receiving training from most sources other groups. Union members generally mained high, in contrast with the de- was usually reduced for 1) those who received higher wages. Large firms paid clining access of British and Australian had gained work experience prior to more. Training had no apparent effect youth to company training as work ex- the completion of schooling, and on the growth of wages, but each pre- perience increased. 2) those with more than three previous vious training episode tended to raise jobs or those having recently changed Although the analyses were confined subsequent wages. Company-based jobs. The researchers concluded this to the study of formal training reported training had the largest effect on wages may have been because youth were by survey respondents, and excluded in the U.S., followed by training from potentially important investments in in- business/technical institutions. For the formal forms of training and learning on

46 Evaluation Forum n Issue 12 n Summer 1997 Evaluation Issues and Activities n International Perspectives on Youth Issues

the job, significant insights were ac- ty of training from schools and other tion of this problem. Meanwhile, tech- quired: off-site sources. nological change is likely to increase this disparity. The study’s conclusions n American youth receive less formal n Less-educated youth receive little training on entry to the labor market formal post-school training and are call for preventive policies and remedial but more formal training with time less likely to receive training in in- programs to reduce dropout rates and in the labor market. dustries experiencing rapid techno- increase academic proficiency in K-12 logical change. education. n In the U.S., employers provide A more specific issue raised by the young workers with training that is n Union membership is associated increasingly company-based, relying with an increased probability of for- study is the comparative feasibility and less on outside sources as workers mal training; however, this may in- effectiveness of company-based train- acquire job tenure — which responds hibit the receipt of informal training. ing and programs such as JTPA that well to technological change. rely mainly on off-site training. Equity n Company-based training had the as well as efficiency is at stake; the n The level of school attainment is an largest positive effect on youth wag- economically disadvantaged have less important predictor of post-school es and in reducing youth unemploy- access to company-based training. training and labor market success. ment. Also, since company-based training is Better-educated youth are consider- sometimes firm-specific and nontrans- n The positive wage effects for formal ably more likely to receive training, training in the U.S. were twice those ferable, the question of whether gov- especially company training, which for Britain and Australia. ernment should subsidize company- has the greatest influence in raising based training is an additional issue of wages and reducing the likelihood of importance. unemployment. Some Implications Given the apparent strong link be- n Technological change plays a critical role in raising educational and skill tween education and training, the high Editor’s Note requirements. dropout rates for American youth, par- ticularly minority youth, pose a serious For those who want more detail, n Technological change increases the problem for policymakers. The gap be- please see: likelihood of receiving company tween the educated and less educated, training, particularly for well-educat- and between whites and minorities, Tan, H. B. Chapman, C. Peterson, and A. ed youth, and reduces the probabili- may be exacerbated without a resolu- Booth. Youth Training in the U.S., Britain, and Australia. Santa Monica, CA: The Rand Corporation, 1991.

Evaluation Forum n Issue 12 n Summer 1997 47 Evaluation Issues and Activities n The Performance Measurement Movement

The Performance Measurement Movement

Editor’s Introduction in a globalized economy, accountability The articles in this series lack the in- for tax money spent has become an in- tegration readers have grown accus- There is nothing new or mystical creasingly desired commodity at all lev- tomed to in Evaluation Forum, but there about the term “performance.” It has a els of government. Accountability in the is a modicum of logic in their selection. familiar and ancient aroma. Adam 1990s is now defined as high perfor- They address different aspects of a Smith surely talked about it. At base, it mance. In the private sector it would in- larger strategic planning process, of is a private sector concept. We talked volve high productivity and the sale of which performance measurement is a about the performance of the Model A products or services at a high level of key dimension. In this sense, total qual- Ford several decades ago, and we want market share. In the public sector it in- ity management, short- and longer- to know about the performance of the volves the achievement of proxies for term goal setting, the application of Toyota Celica in the 1990s. We talked social program goals. goal achievement strategies, the evalu- about the performance of certain ma- Just as the private sector must mon- ation of the implementation of these chine parts in assembly lines during itor and evaluate products and markets, strategies and of their effects, and con- the industrial revolution, and we now government must now monitor the out- tinuous quality improvement are interre- need to determine the performance of comes and longer-term effects of social lated components of comprehensive the computers of this decade. programs. Until the late 1970s, govern- strategic planning. The development of What we have not tended to talk ment focused more on “processes” than performance goals, targets and stan- about is the performance of social in- on the “products” resulting from inter- dards to monitor the progress of this terventions. These initiatives that flow vening in the lives of individuals, planning process and enhance it, and from social policies have been expect- groups and communities in an effort to the evaluation of the efficiency and ef- ed to be successful or unsuccessful, or enhance life quality and stabilize soci- fectiveness of the strategy in meeting in more sophisticated terms their out- ety. The term “performance” is the cho- goals, cannot be — should not be — comes have been judged efficient or ef- sen label for this new emphasis. But analyzed in isolation from the other fective, but we have not talked about strip the name away, and you have components. the performance of programs until quite “program outcomes” and “program im- Consequently, in the first article in recently. We not only transferred this pacts,” old familiar concepts. this series we look at the definitions of, word from the linguistics of the private What is novel is greater attention to and the relationship between, evalua- sector but indulged in transnational bor- measurement issues in developing out- tion and continuous quality improve- rowing — from Europe, Australia and come and impact indicators. That ment. The second article uses a case other industrialized countries. The word seems solidly beneficial. What is also example — educational skills stan- came into its own in two worlds in the new is the development of targets, dards — in developing a standards- 1980s, the private and the public, and standards and their associated rein- driven performance assessment sys- in the 1990s there is a veritable perfor- forcement mechanisms, incentive/sanc- tem for human service initiatives and mance-oriented movement underway in tion systems. The definition of value programs. The level and kind of skills postindustrial governments. here is more controversial. These attainment are important outcomes to A number of nouns find themselves trends represent a renewal of interest be measured in judging the perfor- commonly used in discussions of public in what strategic planning can and mance of education and training pro- performance — performance measure- should mean and assure. That is com- grams, and in studying the longer-term ment, performance indicators, perfor- mendable. But the trends do not neces- employment outcomes of employment- mance targets, performance standards, sarily support the further elaboration relevant initiatives. performance management systems, and use of evaluation research in The third article focuses on the fed- performance measurement systems, studying processes or program effects. eral government’s large-scale effort in performance standards systems, and That is misguided and unfortunate. the 1990s to move the assessment of so on. What does it all mean, and what In this series of articles, we move government activities away from an ex- is the benefit of focusing so much time beyond the reviews concerning perfor- clusive interest in process issues, and and energy on performance? mance management in Issue #11 of toward a judgment of performance (out- The movement toward the measure- Evaluation Forum, but hopefully recall- comes). This little-publicized initiative ment of program outcomes is at the ing the elements of this managerial re- has created an assessment model ap- heart of what we will call the perfor- form detailed in that issue. The key plicable across the federal agencies, mance measurement movement. In an words in those reviews were strategic with significance for strategic planning environment of high government defi- planning, performance measurement, at the state and local levels. It proposes cits and debts and pressures to down- quality assurance, customer satisfac- a performance measurement system size and restructure both the private tion, and continuous program improve- with broad utility, without ignoring the and public sectors to maintain societies ment. You will see most of them again. problem of implementing such a system

48 Evaluation Forum n Issue 12 n Summer 1997 Evaluation Issues and Activities n The Performance Measurement Movement

nn in the real world of politics and bureau- each other’s program areas and com- cracies. Performance posed of program staff who have sought input from stakeholders. This contrasts The fourth article directs our atten- Measurement and with the feedback to managers coming tion to an important piece of national Continuous Quality from external evaluators. The general legislation in 1993 regarding the as- purpose of CQI is to identify quality sessment of government performance, Improvement problems and suggest solutions. While the Government Performance and Re- this purpose holds true for both TQM sults Act (GPRA). It analyzes the imple- and CQI, these approaches are usually mentation of this legislation in ten pilot Performance measurement and con- conceived as separate but complemen- sites. Again, the GPRA suggests a per- tinuous quality improvement seem tary efforts. formance assessment model, and pin- welded together in the two-decade rush The key to the potential relationship points the potential and actual difficul- to focus on the results of social initia- ties in applying it. between evaluation research and CQI, tives, experiments and programs for the the authors propose, lies in the fact that The fifth article reviews the U.S. manifest purpose of producing more employer organizations and social pro- General Accounting Office’s effort to useful information for improving pro- gram organizations committed to CQI serve as a consultant to the Congress gram effects over time. The relationship are significantly different from tradition- on how best to carry out the purposes between these two concepts earmarking al organizations. Those trained in CQI of the GPRA. To play this role, the GAO a new movement in social policy and they say, exhibit these characteristics: studied performance-oriented assess- program assessment on the one hand, ment reform in four industrialized coun- and evaluation research on the other, n They focus on customers’ internal tries. has seldom been addressed directly. and external needs and requirements. This issue is finally given some atten- The sixth and seventh articles in the n They are trained in, and continually series discuss the implications of the tion in the June, 1995 edition of Evalua- practice systems analysis and strate- entrance of government auditors in the tion Practice, a journal of the American gies for identifying quality problems U.S. and Europe into the performance Evaluation Association. The article is and program processes. measurement movement, and the prob- “Implications of Continuous Quality Im- n They use a team approach in study- lem of neglecting evaluation research provement for Program Evaluation and ing, analyzing and carrying out im- in the rush to tie budgeting to simple Evaluators” by Melvin M. Mark, a psy- provements in quality. outcome measurement. chologist at Pennsylvania State College, and Edward Pines of the Industrial En- The eighth and ninth articles give us n They monitor the performance of gineering Department of New Mexico processes in order to assist in the im- a glimpse of the performance measure- State University. ment movements in Australia and the provement of quality, and in validat- Netherlands. ing that improvements in quality What is CQI? have in fact been made. The series concludes with two im- Unlike most analysts, Mark and portant critiques of performance mea- These elements refer largely to the Pines appear to equate continuous qual- characteristics of CQI-oriented staff. surement systems, based on a study of ity improvement (CQI) with total quality their unintended effects. These articles The authors contend that CQI staff ex- management (TQM). Most experts on pect and experience ownership in this suggest that we use caution in using the contemporary performance mea- such systems, and recommend a closer organizational change effort, which dis- surement movement see TQM as a tinguishes them further from traditional linkage between performance measure- team-based management strategy for ment and evaluation research. organizations. However, a major compo- commiting decisionmakers to improving nent of CQI is commitment to this production processes and worker pro- concept on the part of top-level man- ductivity, and for meeting customers’ agement. The overriding goal is im- expectations. TQM has also involved provement of the quality of organiza- regularized monitoring of these process- tional and service delivery processes, es as well as of desired outcomes. CQI which is to result in improved services more often refers to the feeding back of and outcomes. various kinds of assessments of social programs to their management struc- The authors claim that these facets ture, for making ongoing adjustments to of CQI set CQI-oriented organizations the implementation, interventions, tar- apart from traditional ones that tend to get groups and/or outcomes of pro- centralize management power, operate grams. without prescribed procedures for iden- tifying or resolving quality problems, The emphasis in CQI is frequently on and emphasize results more than pro- program implementation — i.e. organi- cesses. The CQI organization requires a zational and service delivery structures more participatory approach to manage- and processes — and the information- ment, organizational development, and fed back to management tends to come service delivery. most often from a team cross-trained in

Evaluation Forum n Issue 12 n Summer 1997 49 Evaluation Issues and Activities n The Performance Measurement Movement

nn Implications for Evaluation important and useful, is not the same as Research scientific evaluation, and does not come Challenges in equipped with the same techniques for The authors propose that the partici- sorting out the cause-effect relationship Developing patory style of CQI-oriented organiza- between interventions, implementation Performance tions makes evaluation planning easier modes, client characteristics and out- to accomplish, and assists evaluators in comes that an application of scientific Measures and implementing certain types of evalua- method affords. tion activities. For example, they say it Standards: The Case aids evaluators in conducting evaluabili- The authors point to certain pro- ty assessments because the organiza- grams that have been the ongoing of Educational Skills tion’s objectives and the processes object of CQI efforts, indicating that through which they are meant to be some of these programs are, at their achieved have been clearly defined. core, inefficient and ineffective by eval- The skills standards project of the Also the design of the program (its the- uation research standards and should be National Governors’ Association (NGA), ory of change), the resources available discontinued instead of “improved.” conducted in cooperation with the Na- to the program, and the causal chain in- And CQI teams may mistake customers’ tional Center for Education and the volved in moving clients from pre-inter- perceptions of increased quality for the Economy, is an important and useful il- vention status to postprogram status real thing. The authors see a need to lustration of some of the challenges in- tends to be specified. In this sense, a evaluate how well CQI approaches are volved in measuring performance. A successful CQI-oriented organization working in terms of actually improving series of reports on performance mea- comes close to precluding the need for programs, using more rigorous evalua- surement have been produced by the an evaluability assessment, the authors tions to help determine their effective- project for use by the National Skills suggest. ness vis-a-vis their stated objectives. Standards Board created in the Goals And process evaluations of CQI-oriented 2000: Educate America Act. This Legis- What Mark and Pines refer to as in- organizations can be very valuable. Also lation mirrors recommendations in the ternal formative evaluation is also as- they see the merging of CQI-generated 1990 report by the Commission on the sisted in a CQI organization, they say, information and information from pro- Skills of the American Workforce. The since in many cases the two approaches cess evaluations as a very useful mar- long-term objective of the Board is to express the same goal. This goal is to riage. establish a framework for a national sys- provide relevant and timely information tem of industry-based skills standards on the basis of which managers can and credentials. make decisions — information that can influence the direction management de- Editor’s Note In this article the focus is on one cisions take. particular NGA report in the series, Per- Readers may be interested in the formance Assessment Systems: Implica- It may be a different story with sum- following related references: tions for a National System of Skill Stan- mative evaluations, they believe, be- Capper, C. A. and M. T. Jamison. “Let the dards. Volume II--Technical Report by cause of the potential for bias in draw- Buyer Beware: Total Quality Management John G. Wirt. Wirt carefully reviews ing causal inferences from program and Educational Research and Practice,” several U.S. Department of Education results that have been produced sub- in Educational Researcher 22, 1983. projects, capturing insights for applica- stantially differently, by a participatory Lipps, G. and P. R. Grant. “A Participatory tion in designing new systems of stan- CQI process versus a traditional one. Method of Assessing Program Imple- dards-driven performance assessment in Without a careful process evaluation mentation,” in Evaluation Practice 13, education. Figure 1 provides an over- (implementation study), they say, the 1990. view of the purposes of these projects. validity and generalizability of outcome Wholey, J. S. “How Evaluation Can Improve ln all three projects the general mission studies can be quite shaky. Agency and Program Performance,” in has been to raise student learning stan- To prevent the production of biased Improving Government Performance ed- dards and reform assessment practices. results, Mark and Pines insist that CQI ited by J. S. Wholey, K. E. Newcomber The measurement concepts used are teams need training in evaluation re- and Associates. San Francisco, CA: Jos- more “judgment-based” — similar to sey-Bass, 1989. search principles and standards, as well those used in occupational skills assess- as cautions in interpreting the results of ment — than “objective,” such as in ed- their efforts. But the largest challenge ucational testing. for evaluators may lie in convincing CQI staff that what they are doing, however Varying Assessment Methods Wirt’s major interest is in the effica- cy of different versions of assessment systems. He claims that research has shown unequivocally that the use of tra- ditional standardized educational tests can narrow the learning environment by encouraging an overemphasis on factual knowledge and lower-level educational skills, even though standardization clearly increases the statistical reliability of performance measurement. Broaden-

50 Evaluation Forum n Issue 12 n Summer 1997 Evaluation Issues and Activities n The Performance Measurement Movement

levels of performance and deciding Figure 1 n Overview of Certain U.S. Department of Education Projects which levels, or combinations of levels, should be used as absolute standards of Project Purposes expected performance. Setting these ab- National Board for Professional To improve the quality of education for all solute standards is what gives standards Learning children by setting national standards for the power to direct behavior. accomplished teachers. The first problem arises when the re- To change the nature of teaching as a ca- sults of assessments are used for “high- reer. stakes, go-no-go” certification. In this To foster ongoing professional conversa- case, the results of assessments must be tions about accomplished teaching. unusually reliable or errors will surely be made. In fact, Wirt says there is am- To influence the way the educational pro- fession views professional development. ple evidence that such mistakes have been made. Another problem is that a To change the public’s perception of teach- single, overall judgment of pass/fail ing. must be made in situations where there are multiple content dimensions for New Standards Project To bring about systemic change in educa- judging performance, multiple perfor- tion through reforms in curriculum, as- sessment and teaching practices. mance exercises for judging perfor- mance, and multiple judges. One solu- To develop internally-benchmarked educa- tion, Wirt suggests, is to create expert tional standards all students should panels that develop performance stan- achieve. dards used mainly as benchmarks in the To create related performance assessment judging process. and examination systems. There is also the problem of general- To replace “standardized, norm-refer- izing the results of performance assess- enced, multiple-choice tests of basic skills ments. However, achieving comparabil- and knowledge that drive classroom prac- ity in judgments throughout a perfor- tice in the wrong direction” with new as- sessment forms based on the perfor- mance assessment system can be in- mance of tasks encouraging in-depth creased, Wirt proposes, by involving thinking and an integration of a range of large numbers of practitioners in the skills. judging process. This may reduce statis- tical reliability but it would increase California Learning Assessment To develop a statewide performance as- comparability in complex performance System sessment system supporting local school systems. reform and improvement. To apply a series of curriculum “frame- Comments on Wirt’s Analysis works,” or standards, in instituting reform. Although the analysis is education- specific, some of the general insights are applicable to the development of ing tests to incorporate higher levels dards. Content standards distinguish skills standards and the measurement of and more complex reasoning skills forc- among different “domains of thought performance in workforce preparation es a reduction in the level of standard- and action” for which standards are to systems as well as in other social pro- ization, making it difficult to generalize be set. gram areas. The dilemmas and prob- test results. Therein lies the dilemma in All three assessment projects have lems identified in the educational field measuring complex skills. relied on methods that present test-tak- have transfer application to employment Wirt advocates for new forms of ers with realistic problem situations and and training and other policy fields. An standards and assessment strategies. judge their responses using panels of important caution can be borrowed These are significantly different than raters who follow a structured method from Wirt’s analysis, namely that a nar- standardized testing, testing based on of evaluation. The measurement as- row set of performance measures rarely the psychological model of skills and sumptions supporting this “interpreta- describes the true complexity of interven- abilities, and the job-competency model tive” model are substantially different tions and outcomes adequately, even based on industrial management con- from those underlying the statistical ap- though the statistical reliability of per- cepts. proach. Wirt believes that the philo- formance judgments using these mea- sophical framework for setting stan- sures — and their generalizability — In this respect, the three case exam- dards and assessing performance in may be greater. Consequently, in the ples offer some common features of in- these projects offers promising alterna- context of growing reliance on perfor- terest. Content standards are described tives for industry skill standards and mance assessment systems, there is a as “texts” presenting a vision of expert other systems. serious need to be vigilant about such performance rather than being extensive problems and seek a better balance in lists of discrete tasks or skills. Concrete developing them across all social policy examples of actual complex perfor- Problems in Setting Standards initiatives. mance are used to communicate both However, Wirt gives attention to two content and performance-level stan- major problems in describing different

Evaluation Forum n Issue 12 n Summer 1997 51 Evaluation Issues and Activities n The Performance Measurement Movement nn ployment programs, the recommenda- Health and Human Services, and the Of- Performance tions include developing more efficient fice of Management and Budget — with and sophisticated data systems and col- the participation of the Departments of Measurement lecting more adequate data for monitor- Agriculture, Commerce, Veterans’ Af- Systems and the ing the progress of program participants fairs, Housing and Urban Development, and the “performance” of programs. Re- representatives of state and local gov- National Performance garding the creation of customer-orient- ernment, and the employer community. ed “One-Stop Centers” intended to pro- Program coordination is seen as a key Review vide a comprehensive set of services task in developing a systematic, ongoing across multiple federal and state pro- national planning process. A number of grams, the recommendations cite three subtasks arc considered critical: In March 1993, President Clinton kinds of information needs: asked Vice-President Gore to lead the n Development of a core set of uniform National Performance Review (NPR), an n Information accessible to customers terms and definitions across the myr- effort to “re-invent” the federal govern- on the labor market, and the quality iad of workforce development pro- ment. Annual reports on the progress of of different education and training grams. options. this initiative were expected. The first n Standardization of fiscal and admin- report of the NPR made no less than n Information on the level and kind of istrative procedures, rules and regula- 384 recommendations. Subsequent re- satisfaction experienced by custom- tions. ports, and Accompanying Reports on ers with the program services they n Development of a complementary set special issues or particular agencies, receive. added to these recommendations. At of results-oriented performance stan- the same time, these reports gauged n Information about the extent to dards. which programs are “performing” progress against NPR’s goals: to reduce n Collection of more useful qualitative vis-a-vis the measures used to judge the size and redefine the role of the fed- information on program perfor- performance. eral government. The underlying pur- mance. pose was to make the central govern- The NPR views “ensuring public ac- NPR views these assignments as im- ment more efficient and effective. countability” as an essential function of portant contributions to “a more com- the federal government in the federal/ In this article we summarize NPR prehensive, universal, customer-driven, state/local partnership supporting the recommendations regarding information integrated workforce development sys- establishment and operation of the One- production and performance measure- tem.” To move closer to such a system, Stop Centers. An appropriate definition ment. One of the major actions taken in it is clear that agencies will need to vol- of accountability, NPR suggests, de- support of NPR’s agenda concerning unteer considerably more staff time in pends on acquiring sufficient informa- these two issues is the development of the context of no new resources. performance agreements signed by the tion on the multiple programs involved, President with the directors of his major and on the development of adequate in- federal agencies. These agreements in- formation systems for the collection, The Development of a Core volve the development of measures of storage, extraction and analysis of these Set of Uniform Terms performance for each agency. By 1998, data. and Definitions the federal government is expected to To encourage states to develop state- This unique and difficult initiative is produce its first results-oriented finan- wide workforce development systems expected to produce substantial results: cial statement. In this sense, perfor- and local workforce boards that more ef- n Joint intake forms. mance measurement and budgeting are fectively coordinate policy development, to be closely coordinated for the first planning and oversight, the NPR recom- n Common data elements. time. mends that federal funding to initiate n Sophisticated computer technologies One-Stop Centers be available only to to improve information, services and The 1993 NPR Accompanying states willing to increase accountability. access. Report Directed to the U.S. NPR also suggests that the Department n Comparisons across related pro- Department of Labor of Labor provide state workforce coun- cils with resources for capacity-building grams. In the National Performance Re- and information on “best practices.” Per- view’s series of reports, twenty-one rec- In the 1992 JTPA Amendments, the formance-based competition across One- ommendations were directed to the U.S. Department of Labor was required to Stop Centers is also promoted. Department of Labor. Staff teams inter- establish a federal interagency work nal to the Department were expected to group that would identify a core set of participate with the NPR in implement- Strategic Planning consistently defined data elements and ing these recommendations. Although The 1993 report essentially creates a definitions for JTPA, JOBS, the Perkins the suggestions focus mainly on cutting new environment for national strategic Act, and Title V of the Older Americans costs and red tape, consolidating select- planning regarding workforce develop- Act. ed programs, and integrating services, ment. The National Economic Council is The report of this task force, Core they also give attention to information expected to establish a strategic plan Data Elements and Common Definitions needs and accountability issues. With through the joint efforts of the Depart- for Employment and Training Programs respect to the consolidation of reem- ments of Labor, Education, Energy, was published by the U.S. Department

52 Evaluation Forum n Issue 12 n Summer 1997 Evaluation Issues and Activities n The Performance Measurement Movement

of Labor in January 1996. (Issue #11 of The 1993 NPR Accompanying ed early-on in the public sector — in Evaluation Forum reviewed this report.) Report on Mission-Driven, the government performance movement This report also supports the creation of of the 1940s and 1950s, the Program- an interagency Workforce Development Results-Oriented Budgeting ming, Planning and Budgeting frame- Council at the federal level. Tying the budgeting process to per- work emerging in the 1960s, the Zero- formance measurement, NPR defines Based Budgeting of the 1970s, and the the functions of budgeting broadly: The Development of a Management By Objectives and Total Quality Management concepts of the Complementary Set of Results- n A managerial process that determines 1980s. Oriented Performance the direction that should be taken in Standards improving management and overall In the private sector, major corpora- efficiency. A set of desired outcomes for most tions restructuring for global competi- tion have reoriented their own priorities workforce development programs has n A marketing process that helps pro- already been identified, but the defini- mote programs and services. toward customer service, placing prima- tions of these outcomes often vary, and ry emphasis on quality and requiring at- n An accountability process through tention to continuous improvement in not all relevant outcomes are given at- which policy priorities and desired tention. Furthermore, the quality of processes and products. Clearly bench- outcomes can be developed and marking, or measuring performance these outcomes is frequently ignored. monitored. More significantly, performance stan- against a standard, has come into its dards may involve too few variables and Having so defined budgeting, the own. The public and private sectors are measures, or fail to incorporate the most NPR concludes that traditional federal converging in embracing results. Which critical ones. Evidence indicates that government budgeting has neglected sector has had the most influence on standards, as a reward and sanction goal-setting and overemphasized inputs, the other is uncertain but a 1993 survey system affecting program funding, can reducing government accountability. of corporate executives in 75 major redirect programs away from their in- The NPR claims that a change in the American corporations, conducted by tended goals and target groups. And culture of budgeting can “encourage the U.S. Treasury Department, conclud- standards may not require sufficient in- long-term thinking, define goals, trans- ed that federal program managers formation about the level of achieve- late them into desire results, and use should learn from “corporate best prac- ment of key outcomes to be of maxi- performance measures more effectively tices,” referring to the growing interest mum fairness and use in planning and to make more informed decisions on in outcomes and their quality. budgeting. program priorities and resource Internationally, postindustrial coun- allocation...[and to] convert account- The interagency work group is ex- tries in Europe and Asia, also feeling the ability for spending money to account- bite of broader competition, have in- pected to walk this mine field success- ability for achieving results, within the fully, developing a set of performance creased their use of performance man- resources allocated, to meet or exceed agement to guide public administration, standards that represents the common performance standards and objectives.” desired outcomes for multiple programs, program management and budgeting. recognizing that not all programs We have entered a new era in which should be guided by the same stan- History satisfying the taxpayer and the custom- dards, and that all standards require on- Actually there is nothing new about er has moved to center stage. This is re- going adjustment based on evaluations the concept of results-oriented financial flected in the landmark Government Per- and program experience. planning. It is standard in academic formance and Results Act of 1993. The planning departments, as part of strate- purposes of the Act are outlined in Fig- ure 1. The Development of Qualitative gic planning principles. It was anticipat- Information on Program Performance Figure 1 n Purposes of the Government Performance and Results Act Although the NPR report for the De- partment of Labor does not elaborate on n To hold federal agencies accountable for achieving program results. what is meant by “qualitative” out- n To initiate a reform of performance through a series of pilot projects in federal agencies comes, we can assume that attitudinal which test performance management concepts prior to full implementation of the Act — measures of customer satisfaction are i.e. set goals, measure performance against these goals, and report publicly on pro- an example. The larger goal seems to be grams. an expansion of the set of desired out- n To promote a new emphasis on results, on service quality, and on customer satisfac- comes to include information about se- tion. lected characteristics of these outcomes, n such as the nature of the training re- To require federal managers to plan how to meet objectives and provide them with in- ceived, the occupational category of the formation about program results and service quality. job recipients obtain, the nature of the n To provide more objective information on achieving statutory objectives, and on the rel- work environment and the characteris- ative effectiveness and efficiency of federal programs and spending. tics of the workplace. All agencies are to 1) define their long-term goals, 2) set specific annual performance tar- gets, and 3) report annually on the performance vis-a-vis the targets.

Evaluation Forum n Issue 12 n Summer 1997 53 Evaluation Issues and Activities n The Performance Measurement Movement

The interest in “re-inventing” gov- n The streamlining and improving of goals and objectives, drawing on pol- ernment does not necessarily mean that the budget development process, to icies and program legislation, histori- government has gone beyond the sheer provide managers with more timely cal and current program data, and rhetoric of the new accountability move- information on policy priorities and the needs and expectations of the ment. A 1992 GAO report, Program Per- funding levels, and to coordinate program’s customers and stakehold- formance Measures: Federal Agency Col- budget resources with agency mis- ers. lection and Use of Performance Data, sions, goals and results. n Identification of the kinds and quan- indicated that a survey of federal agen- These three initiatives were in sync tity of goods and services to be pro- cies and their departments showed that with the Government Performance and duced and for whom, and the quality fewer than half had actually used per- Results Act in 1) developing strategic expected in those goods and servic- formance information in monitoring plans, 2) establishing performance es. their progress in achieving agency goals, agreements, and 3) creating results-ori- even though two-thirds had developed n A comparison of programs with simi- ented task forces to coordinate the de- strategic plans and were collecting per- lar ones, or with examples from the velopment and use of performance mea- formance data. Even fewer were using private sector, to establish perfor- sures, ensuring that performance performance measures in reports dis- mance benchmarks, based on the ac- information was reliable and valid. But seminated to other agencies and the complishments of other programs. the NPR’s recommendations went fur- public. More significantly, few were us- ther: n Development of operational plans ing performance information in making and shorter-term performance targets policy and management decisions. The n Each agency is to compare well with or standards for reaching the bench- existence of NPR set off a siren for other similar organizations in achiev- marks, based on evaluations of cur- change. ing “benchmark standards for quali- rent activities and efforts to improve ty, service and cost.” them. NPR’s Recommendations n All directors, managers and budget n Establishing effective agency teams offices are to incorporate perfor- Acknowledging that unacceptably to support the above, and to oversee mance objectives and results as key high federal deficits have spurred a the monitoring of progress. reconsideration of the budgeting para- management elements in developing, digm internationally, the NPR is never- reviewing and deciding on budgets. n Development of common goals for theless insistent that its recommenda- data collection across programs, such n Appropriation accounts and line as common outcome measures and tions can apply equally well in a period items are to be aligned with the pro- benchmarks across programs with of abundance. These recommendations grams designed to carry out policies, similar purposes, interventions and fall into several categories: and are to be charged with the costs target populations. n The strengthening of accountability these programs generate. for results, with politicians defining n Inclusion of performance objectives n Agency heads are to adjust their op- and results as key elements in finan- the political priorities, followed by erating plans to clarify performance cial and managerial reviews. agreements with managers about goals. goals and how achievement of these goals is to be measured. n The President is to propose biennial Challenges in Implementing budgets, and Congress is to adopt bi- NPR’s Recommendations n A number of postindustrial countries ennial budget resolutions and appro- are now requiring the development priations. Although there is now considerable of mutually agreed-upon missions, agreement in government that it is es- goals and objectives for organization- n The Administration is to establish a sential to measure and monitor perfor- al performance, which tie the perfor- coordinated team approach and de- mance, the development of performance mance of senior government officials velop an internal mechanism for al- measures and standards, and particular- to a government’s major goals. Many locating funding according to broad ly the notion of linking these with bud- corporations are doing the same, government missions, agency by geting, pose some problems. NPR is re- linking performance agreements with agency. freshingly frank about them. organizational goals and team perfor- Such suggestions are tied to the Performance measurement needs to mance. In the U.S., effectively imple- NPR’s major interest in results, quality menting the Government Perfor- be simple and useful, the NPR report and customer service, with the logical recognizes, with the goal of using a mance and Results Act is a case chronology of steps similar to these: example of a much broader trend. small number of easy to define mea- sures. Developing performance mea- n Development of an agency’s mission. n Once decisions have been made and sures is to involve living with a particu- agreements established, the empow- n Clarification of an agency’s goals and lar set of assumptions about what is ering of managers to achieve their objectives, consistent with its mis- most important to achieve, and then goals, by providing sufficient resourc- sion. making choices among competing long es and reducing excessive regulatory vs. short-term goals. This is to be ac- n Description of the strategies the burdens. agency plans to use in achieving the complished in the context of the cost of

54 Evaluation Forum n Issue 12 n Summer 1997 Evaluation Issues and Activities n The Performance Measurement Movement

acquiring good information about the It also requested that ten pilot levels, earnings growth and share pric- achievement of those goals and the hu- projects be developed by the federal es. This cross-cultural transfer required man resources needed to interpret that agencies to test the utility of this new that government organizations under- information for policy use. legislation. Since 1993, no less than sev- stand as clearly as successful businesses enty projects have been created. For the the goals toward which activity should Cost and resource considerations can first time, twenty-four major federal be directed. Now, in 1996, federal agen- narrow the set of variables utilized. Not agencies have been required to provide cy-White House performance agree- all critical elements of a program may annual audited financial statements. ments identify each agency’s perfor- be measured, risking an insufficient un- mance goals and objectives, document a derstanding of its full complexity. Fur- The Act essentially set the stage for commitment to honor them, and seek to thermore, the measures chosen may not the National Performance Review. But reward senior agency personnel on that be reliable or valid. Also different feder- Gore points out that the NPR was not basis. al programs have significantly different the only effort at government reform in desired outcomes, therefore perfor- the 1990s. He reviews a host of legisla- The linguistics of the NPR and relat- mance must be measured differently in tive initiatives that provide additional ed efforts to reform government is rife diverse programs. evidence of a new awareness of the im- with downsizing, streamlining, restruc- portance of “government accountabili- turing and privatizing. Government ac- Unlike previous accountability ty,” such as the Government Manage- tion taken since 1993 and reported by thrusts, the new movement requires ment Reform Act, the Chief Financial the NPR would suggest that a great deal considerable customer feedback in de- Officers Act, and the Federal Acquisi- has been accomplished by this combi- veloping performance measures. Admi- tions Streamlining Act. And other forces nation of strategies. Gore compares the rable and useful as this may be, it can have been at work simultaneously, at NPR approach to enterprise develop- slow and complicate the process. It both the federal and state levels, which ment, suggesting that a good enterprise sometimes risks divisiveness that reduc- supported NPR’s reinvention strategy — measures what matters, and these mea- es an acceptance of the measures and such as a growing interest in studying sures tell managers how and in what the planning process. And the move- customer satisfaction. NPR built on this ways they are or are not succeeding as ment to focus more exclusively on re- interest by establishing Customer Service well as what to do to improve perfor- sults may create “performance informa- Standards, which amounted to perfor- mance. “If the essence of enterprise de- tion bureaucracies” that develop a life mance standards specific to customer velopment is risk-taking and experimen- of their own. The NPR does not ignore satisfaction. The standards involved tation,” Gore says, “measurement tells such challenges. specific outcomes and commitments for us how the experiment is going.” achieving them. The 1995 NPR Report A parallel effort developed bench- Some Concluding Review This time the NPR report was in marks for service quality, drawing Comments book form, written by the Vice-Presi- insights from private/public “best prac- dent — a journalist before moving into The National Performance Review tices.” To lend strength to the accumula- has maintained faithfulness to its mis- politics. This most recent report is an tive effect of different reform efforts, in interesting review of progress in imple- sion, to recommend and monitor the March 1995 President Clinton issued a creation of a new form of central gov- menting previous recommendations, but directive reinforcing the requirement it does not neglect ways in which the ernment. The principles and standards that agencies develop and publish cus- it seeks to promote are becoming the federal agencies should resolve continu- tomer standards. He required them to ing problems. Gore reiterates that there “universals” for postindustrial govern- survey customers to identify their ments. What now seems to be needed, was a tendency in the past for the feder- needs, measure customer satisfaction al government to collect information on as part of NPR’s activities, is a perma- based on what outcomes mattered most nent task force to study the risks and inputs to the neglect of outcomes, and to the customers themselves, and inte- with little attention to the quality of ei- benefits of moving in the results-oriented grate customer service activities with all direction, giving careful attention to the ther. He claims that agencies often for- other performance initiatives. Among got what their initial missions and goals potential risks and monitoring the other purposes, this approach was ex- tradeoff over time. were, in the context of lax expectations pected to increase managers’ ability to about understanding the results of their continuously improve program organiza- This task force should also address efforts. tions, services and outcomes. the issue of what role evaluation re- search should play in this new move- Gore reminds us that the 1993 Gov- What the NPR was attempting to do ernment Performance and Results Act ment. Science-based program evaluation was to infuse government with what it is mentioned in passing, and not nega- was a way to guide agencies away from felt was the best version of private sec- a focus on procedures and processes to- tively, but it is not made an integral part tor concepts and practices, to improve of the performance movement. Without ward an emphasis on performance. The its capability, responsiveness, and per- Act required agencies to define their evaluation research, we will not know formance. The public had become the the policy significance of the “results” goals more precisely, make specific stockholders, the government the corpo- commitments to achieving them, and we have so compulsively measured and rate manager, program services the reported in the 1990s. establish formal “outcome measurement product, and performance measurement systems” to allow them to track the way to monitor market share, sales progress against goals.

Evaluation Forum n Issue 12 n Summer 1997 55 Evaluation Issues and Activities n The Performance Measurement Movement

nn Kamensky, J. and J. Tessauro. Performance Editor’s Note Measurement in the United Kingdom and Testing the Australia. Paper presented at the Perfor- The National Performance Review mance Measurement Seminar. Washing- Government has published yearly progress reports ton, D.C.: National Academy of Public based on its mission and goals, and a Administration, May 1992. Performance and large number of Accompanying Re- Kettl, D. Improving Government Perfor- Results Act ports. mance: An Owner’s Manual. Washington, Annual Reports D.C.: The Brookings Institution, 1993. The National Academy of Public Ad- ministration (NAPA) was chartered by September 1993: From Red Tape to Results: Moore, M. H. Accounting for Change: Rec- the U.S. Congress in 1967 to improve onciling the Demands for Accountability Creating a Government That Works Bet- the art and science of public gover- ter and Costs Less and Innovation in the Public Sector. Washington, D.C.: Council for Excellence nance. It is composed of four hundred September 1994: Creating a Government in Government, 1993. current and former federal Cabinet of- That Works Better and Costs Less: Sta- ficers, members of Congress, governors, tus Report Oregon Benchmarks: Standards for Measur- ing Statewide Progress and Government mayors, state legislators, jurists, public September 1994: Putting Customers First: Performance. Oregon Progress Board, managers, business executives and aca- Standards for Serving the American Peo- December 1992. demicians. In 1994, NAPA formed an ple. Osborne, D. and T. Gaebler. Reinventing Advisory Panel on Improving Govern- September 1995: Common Sense Govern- Government: How the Entrepreneurial ment Performance. This panel was co- ment: Works Better and Costs Less, by Spirit is Transforming the Public Sector. chaired by Harry Hatry of The Urban Al Gore. Reading, MA: Addison-Wesley, 1992. Institute and Joseph Wholey of the Uni- versity of Southern California. Wholey Accompanying Reports Schick A. Five Reforms in Search of Budget is now at the President’s Office of Man- Report for the U.S. Department of Labor Control: Congress Versus the Federal Budget. Washington, D.C.: Congressional agement and Budget (OMB). The results Transforming Organizational Structures Research Service, January 1992. of the panel’ s work were reported in a Improving Customer Service U.S. Department of Treasury: Performance monograph published in 1994, Toward Mission-Driven, Results-Oriented Budgeting Measurement: Report on a Survey of Pri- Useful Performance Measurement: Les- sons Learned from Initial Pilot Perfor- Reinventing Human Resources vate Sector Performance Measures. Washington, D.C., January 1993. mance Plans Prepared Under the Govern- Management Reengineering Through Infor- U.S. General Accounting Office: Managing ment Performance and Results Act. mation Technology for Results: Experience in Four Europe- In the preceding article about the Rethinking Program Design an Countries and How They Might Be President’s National Performance Re- Strengthening the Partnership in Intergov- Applied in the U.S. Washington, D.C., view, we discussed the Government Per- ernmental Service Delivery January 1993. formance and Results Act of 1993 Additional References of Interest ______. Using Performance Measures (GPRA). This legislation requires a pro- in the Federal Budget Process. Washing- cess of goal setting, performance mea- Epstein, P. “Get Ready: The Time for Perfor- ton, D.C., July 1993. mance Measurement is Finally Coming’” surement and regularized review as a in Public Administration Review, Septem- Volcker, P. A. Leadership for America: Re- basis for a new performance-oriented ber/October 1992. building the Public Service. Washington, approach to public governance — all in- D.C.: The Federal Reserve, 1989. gredients in a new definition of strategic planning. Ten pilot projects were to test this approach over a three year period to provide insights for implementing the Act. However, no less than seventy-one performance plans were submitted to OMB. This involved thirteen of the four- teen federal departments. Fifty-two plans were accepted. Quite remarkably, this occurred in the context of minimal OMB guidelines, no training, and no ad- ditional funding for this large project. NAPA had taken a strong interest in performance measurement since 1979 when it published the Productivity Im- provement Handbook for State and Lo- cal Government. Subsequently the orga- nization passed a resolution supporting performance measurement and monitor- ing. Later NAPA established the Adviso- ry Panel to review the performance plans of the pilot projects testing the GPRA, and to synthesize ideas from this unprecedented planning effort. This ar- ticle summarizes NAPA’s review.

56 Evaluation Forum n Issue 12 n Summer 1997 Evaluation Issues and Activities n The Performance Measurement Movement

Review Criteria observed outcomes could be attributed information was to be communicated to exclusively to programs. Interestingly, program managers for program im- NAPA reviewed forty of the perfor- agency officials proposed that only in- provement, a major purpose of the mance plans for pilot projects within a depth program evaluations were capable GPRA, the panel found plans unclear particular set of assumptions about per- of establishing this kind of cause-effect about the basis for selecting perfor- formance measurement. The Panel’s relationship, and therefore only when mance targets. In many instances, no perspective was that the measurement evaluations were linked with perfor- targets were proposed, in others the tar- of program “outputs” and “outcomes” mance management could they feel gets chosen appeared not to have in- provides essential information about comfortable about the proposed new volved a re-evaluation of traditional ap- goal achievement — that is, about prob- approach. proaches. Most serious, there was little lems needing resolution and how to im- evidence that performance information Organizational responsibility for prove programs. Performance data were would actually be used to improve per- performance measurement. Participa- also seen as establishing public account- formance — i.e. would be disseminated tion in developing performance mea- ability. The review of the projects to operating units of programs. Further- sures and targets by all levels of agency looked for strategies being used that more, most plans did not suggest disag- management — particularly by top-level would test these assumptions. gregating performance data — such as administrators — appeared lacking. by type of clients, organizational unit, Therefore, the Panel was careful to This weakened agencies’ commitment or geographical area. specify criteria for judging the plans to performance plans. Also, few agen- submitted by the pilot projects. The cies had given thought to performance Explaining program performance. main criteria were: measurement regarding their relation- Plans did not propose ways to include n The reasonableness, clarity and com- ships with other agencies with similar explanatory information in program re- prehensiveness of mission state- missions. ports that would suggest potential rea- sons for a particular level of perfor- ments, and the relationship of mis- Relationship between performance mance on measures and targets. sions to performance plans. indicators, legislative goals and agen- n The mix of performance indicators cy missions. There was insufficient Dissemination of performance in- suggested in the plans, such as mea- linkage between the measures devel- formation to the Congress and public. sures of inputs, processes, outputs oped and the legislative purposes, mis- Performance measures and targets were and outcomes, as well as efficiency sions, and strategic plans of agencies. In often presented in plans without an ad- and service quality measures — but some cases, mission statements were equate context in terms of the level of with the main emphasis on out- entirely too broad to reveal these con- understanding of Congressional leaders comes. nections; in others no strategic plan was and the public. The larger design of evident. There was a general tendency programs, of which performance mea- n The adequacy of plans for judging to rely on existing administrative data in surement was a part, was not well com- data reliability and validity. developing measures, with too little em- municated or explained in commonly n Whether the use of numerical perfor- phasis on what measures might be best understood language. mance targets based on performance in monitoring agency activities against indicators were suggested, and what their missions and goals. Few agencies The Panel’s Recommendations the basis was for these targets. had involved their clients in providing ideas about performance measures. The Panel actively considered recom- n The likely utility of the performance mendations for improving performance measures and targets suggested in Nature and complexity of the per- plans. Figure 1 (page 58) summarizes plans to policymakers and program formance measures. Likely due to the their main suggestions. managers in improving program per- lack of guidelines by OMB, agencies dif- The initial report card on the pilot formance, and to Congress and the fered in their definitions of performance projects does not translate to an “A,” public in demonstrating program ac- measures. Also, the sets of measures but much progress has been made in a countability. were frequently narrow and limited, re- peating the earlier focus on process very short time by large-scale govern- ment bureaucracies that historically re- The Results of the Panel’s Review data. This was so even though most agencies were collecting outcome data. sist change. Given the time line, it sure- Applying these criteria, the Panel’s The lack of customer feedback was evi- ly was not realistic for OMB or NAPA to review produced these conclusions: dent in the paucity of measures for have expected more at this juncture. judging customer satisfaction. Meanwhile, the lessons gleaned in the Timing. Too little time was made NAPA review will contribute important available to federal departments and Validation of data sources. Few per- insights for next steps in this long- agencies for preparing performance formance plans included methods for term, complex and promising federal plans. Personnel in the pilot projects assuring that the data collected for per- project.The GPRA requires all federal clearly needed training in performance formance management purposes were agencies to participate in strategic plan- measurement, as had been recommend- reliable and valid. Where these issues ning and performance measurement by ed in the Act. were addressed in plans, only simple au- fiscal 1999. Performance targets are to Scope. Some agencies had avoided dit strategies, rather than scientific be provided by September 1997 and an- developing performance indicators be- methods, were suggested. nual performance reports are to begin cause they felt they had little control Improving program performance. by March 2000. This is a very ambitious over the influences to be measured, and In reviewing plans for how performance planning agenda. that performance based on targets would not explain the extent to which

Evaluation Forum n Issue 12 n Summer 1997 57 Evaluation Issues and Activities n The Performance Measurement Movement

Figure 1 n NAPA Advisory Panel’s Recommendations Issue Recommendations Timing and Scope of the Performance All departments and agencies should speed the development of of the strategic plans, the Plans development of performance measures, and the identification of sources of information to support the measures. Training in developing performance measures for program improvement purposes needs to be provided, as well as opportunities for sharing information on this task across federal de- partments. Providing explanatory information about the probable relationship between programs’ inter- ventions and their performance should be an integral part of performance reports, so that outcomes likely attributable to programs can be distinguished from those likely due to other factors. The limitations of performance measurement should be made explicit.

Departmental Responsibility for The top level of departments and agencies should play a more active role in strategic plan- Performance Measurement ing and performance measurement, particularly in formulating and improving performance management systems. Program managers should play a strong role in identifying performance measures that will help them improve programs. And the federal government should develop a method for studying the range of agency and program performance measures, to identify and reconcile interrelated missions and measures.

The Use of Performance Measurement Consistent linkages should be evident between missions, goals, strategic plans, and annual performance plans — and these processes and products should be connected logically to the broader goals of top management and the Congress.

Management Systems to Achieve Each agency should establish a hierarchy of goals and performance measures appropriate Legislative Goals, Agency Missions, to the different levels of government, which are internally consistent. and Program Improvement If new data collection appears essential, agencies and programs should propose the addi- tional measures needed. In the process of developing measures, agencies and programs should seek input from their organizational stakeholders and clients which can be incorporated in performance measure- ment systems. Annual performance targets should be set realistically, but should be developed to encour- age progress beyond traditional levels. Consequently, performance plans should include long-term performance goals as well as the targets for the following year. The definition of “excellent performance” needs to be incorporated.

The Nature of Performance Indicators Top agency management should encourage a focus away from the exclusive use of process indicators to the use of a broad range of performance indicators. A consistent set of measures, their operational definitions, and guidelines for applying them should be classified usefully, such as by performance indicators, performance targets, in- puts, outputs, outcomes and resources required. Performance plans should identify basic data sources and what methods will be used to as- sure that measures are valid in representing key factors, are reliably collected, and are time- ly. Technical assistance needs to be provided to maintain the quality and relevance of auto- mated performance-based information systems.

Reporting Performance Classifying information will assist different groups of organizational users at the federal level in fully utilizing performance reports. Simple diagrams can clarify program operations, and illustrative cases can help users un- derstand the program context.

58 Evaluation Forum n Issue 12 n Summer 1997 Evaluation Issues and Activities n The Performance Measurement Movement

nn tion. The associated need to provide Editor’s Note The Experiences of more useful information about the effi- ciency and effectiveness of government For more information on perfor- Other Countries with departments, agencies and programs mance measurement issues, please moved reforms away from a focus on see these publications: Performance inputs and processes, and in the direc- Hatry, H. et al. Performance Measurement - Measurement tion of outcome measurement. A Guide for Local Elected Officials. Washington, D.C.: The Urban Institute, As a tradeoff for increased account- 1981. ability through outcome measurement, The U.S. General Accounting Office typically managers in these countries Hatry, H. and C. Wye. “Evaluation — It’s Po- (GAO) has produced a series of reports have been given greater flexibility and tential Utility for Operating Managers,” in on management reforms that involve Organization Excellence edited by J. autonomy in allocating resources in performance measurement.This article newly decentralized and devolved social Wholey. Lexington, MA: D.C. Heath and concentrates on Managing for Results: Co, 1987. service environments. However, the re- Experiences Abroad Suggest Insights for forms bind regional and local govern- Hatry, H. “Determining the Effectiveness of Federal Management Reforms, published Government Services,” in Handbook of ment organizations to central govern- in May, 1995. The report responds to a ments through multi-level performance Public Administration edited by H. Perry. Congressional request for ideas about San Francisco, CA: Jossey-Bass, 1989. agreements. These assure that managers implementing and assessing the man- will be held accountable for the results Hatry, H. and D. Fountain. Service Efforts agement reforms required by the Gov- and Accomplishments Information: Its they achieve, based on agreed upon per- ernment Performance and Results Act of formance goals. In these agreements, Time Has Come. Norwalk, CT: Govern- 1993. It analyzes the experience of four mental Accounting Standards Board, the definitional steps from mission to 1990. industrialized countries recognized as goals to objectives to measures of perfor- leaders in the movement toward “re- Hatry, H. and D. Fisk “Measuring Productivi- mance are to be a logical progression, sults-oriented” government reform — each step leading to increasingly more ty in the Public Sector,” in Public Produc- Australia, Canada, New Zealand and the tivity Handbook edited by M. Holzer. New specific understandings of the meaning York, NY: Marcel Dekker, Inc., 1992. United Kingdom. of efficiency and effectiveness. Hatry, H. et al. “Eleven Ways to Make Perfor- The Reform Concept and There has been some confusion mance Measurement More Useful to among managers, however, about the Public Managers,” in Public Manage- Its Trade-Offs difference between devolution, which ment, September 1994. The major features of the reforms the GAO defines as the transfer of deci- Wholey, J. and J. Newcomer. Handbook of studied in these countries by the GAO sionmaking capacity from higher orga- Practical Program Evaluation.San Fran- are strategic planning, the development nizational levels to lower, and decentral- cisco, CA: Jossey-Bass, 1994. of goals and objectives, the measure- ization, which the researchers define as ment of progress in meeting these ends, the redistribution of functions from cen- and the reporting of progress to the pub- tral organizational units to more dis- lic. Although “management by objec- persed ones. Some managers have de- tives” thinking is not in any sense new, fined the new flexibility in both many Americans caught up in our cur- directions, irrespective of the intent of rent performance measurement move- the reforms. ment may be unaware that the four countries examined have been carrying Some Useful Conclusions from out results-focused management chang- es for over a decade. This effort has par- the GAO Study alleled their pursuit of changes in the The GAO researchers found these organizational culture of private sector characteristics to describe the move- entities. ment toward results-oriented manage- ment in the four countries studied. It is not surprising that one of the most challenging tasks in instituting n Development and definition of a performance management systems in logically consistent set of depart- these countries has been changing the ment, agency and program missions, managerial culture. There have been goals, objectives and performance substantial differences in the speed with measures. which the desired cultural changes have n Holding management accountable occurred, and in the extensiveness of for desired results. the changes across diverse governmen- tal departments and agencies within n Provision of greater management au- these countries. tonomy in allocating resources with- in overall budget ceilings, and in de- In all four nations, the reforms were veloping incentives for meeting catalyzed by serious economic problems goals. and pressures to provide better evidence of goal achievement, the same influenc- n Capacity-building and technical as- es fueling our own managerial revolu- sistance to support results-oriented management.

Evaluation Forum n Issue 12 n Summer 1997 59 Evaluation Issues and Activities n The Performance Measurement Movement

n Development of information systems views with officials suggested the fol- ing provided to lower levels of manage- to collect, report and analyze depart- lowing: ment. Private sector “market” concepts ment, agency and program data for and strategies were being introduced to n Performance measures need to flow monitoring and assessment purpos- create a more competitive interdepart- directly from a program’s objectives, es. mental, cross-program culture, rein- irrespective of whether they repre- forced by financial and other incentives. sent short-term or longer-term ef- It is the GAO’s conclusion that the ap- Participatory Planning and fects. Implementation plication of these market-oriented ideas n Performance measurement systems has, in fact, made government activity In the United Kingdom and Australia, should involve a small set of key more efficient and effective. But formal management and staff are sometimes measures — i.e. those considered evaluations indicate that there are risks brought into the strategic planning pro- critical to performance. involved in increasing managerial au- cess, leading to a greater sense of own- tonomy, such as a loss of control by n Measures should represent outcomes ership of goals and a stronger commit- central governments, and it is clear that that realistically can be achieved by ment to improving performance. Annual top management has in some cases only a program’s management. operating plans translate strategic plans halfheartedly cooperated with the re- into detailed objectives for day-to-day n Program staff should play a role in forms. activities. In developing such plans, identifying measures. New Zealand and the United Kingdom have emphasized the more immediate n The range of performance measures Supports for Performance measures of performance — i.e. mea- should assure balance, to preclude Systems sures of quantity, quality, efficiency and an overreliance on any one measure. Meanwhile, long-term investments in “output” cost — rather than longer-term information systems and training have n Performance systems should include effectiveness measures. But all four gov- both qualitative and quantitative in- supported reform. In all four countries ernments use a combination of qualita- formation, on both process and out- there has been a recognition that high- tive assessments, customer surveys and comes. er-level management needs to assist more formal evaluations to estimate the lower-level managers in understanding effects of programs. There is a strong n Performance results should be ex- the value of the reforms and in imple- sense of fairness in protecting managers plained, to the extent possible, not menting them — in particular strategic from being blamed for outcomes heavily simply reported. and operational planning, performance influenced by economic conditions and measurement and budget flexibility. n Performance systems should provide other phenomena beyond their ability to more detailed information on the alter. achievement of measurable objec- Performance Reporting and tives regarding factors considered Utilization Mixed-Method Assessment outside the control of program man- The United Kingdom and Canada In Australia and Canada, both perfor- agement as well as influences viewed routinely develop and publish perfor- mance measurement and program evalu- as being under management control. mance targets and standards for depart- ation are viewed as integral parts of n Performance information should be ments and agencies directly serving the management, a perspective we still made available to the public and to public, and report achievements to the struggle to create in the U.S. In Austra- all levels of the managerial hierarchy. public. The United Kingdom’ s Citizen lia, departments are required to evaluate Charter and Canada’s Service Standards the effectiveness of each of their pro- Performance Measurement Reforms require agencies to publish ser- grams every three to five years. In Cana- vice standards, measure performance da also, departments are expected to be Activities against these standards, and request capable of evaluating programs for their The United Kingdom and Canada feedback from the public. Australia and effectiveness. Both “ongoing effective- measure the quality of services and re- New Zealand place less emphasis on ness measurement” for monitoring pur- port results to the public on a regular providing customers with program in- poses and “periodic studies of program basis. All four countries utilize perfor- formation and encourage less customer effectiveness” are central to the new re- mance agreements between different feedback, but they include many service forms. Monitoring involves collecting governmental levels which incorporate quality measures in their performance and reporting information on a set of performance targets. And all four re- systems. These systems also involve rec- outcomes; evaluation involves a more quire departments to make performance ommendations for resolving perfor- in-depth review of the fit between pro- reports available to their central govern- mance problems. gram objectives and outcomes. The use ments and the public. The GAO found, of comparison or control groups is rec- however, that the use of these reports The use made of performance re- ommended, but program evaluations by central governments has been mini- ports has been uneven, however. The more often rely on surveys of program mal, and that efforts are underway to Canadian Parliament found in 1992 that recipients, particularly regarding cus- increase utilization. department performance reports did not tomer satisfaction. provide the kinds of information most In these countries, managers were needed by decisionmakers, did not link being given much more freedom to outcomes with costs, and did not sum- Feedback from Internal make their own spending decisions and marize and interpret performance clear- Reviews and Levels of develop their own incentive systems for ly. The lack of staff, time and expertise Management rewarding high performance. More of often eroded the utility of performance The GAO researchers’ review of gov- this kind of flexibility was, in turn, be- information to potential users. ernment evaluations and their inter-

60 Evaluation Forum n Issue 12 n Summer 1997 Evaluation Issues and Activities n The Performance Measurement Movement

nn Some Comments program rather than to other influences Performance and forces. The outcomes selected are This GAO study reinforces the con- not necessarily those selected by an clusions of other expert reviews of the Auditing and agency or program manager or an eval- performance measurement movement in Evaluation Research uator, since they emerge from a fiscal the industrialized countries, namely auditing, not a program-knowledgeable that there are positive benefits in in- environment. creasing government attention to the short-term and longer-term outcomes Evaluation researchers are beginning Most problematic is the tying of per- and effects of departmental, agency and to feel the invasion of performance au- formance audits to budgeting decisions. program activity. However, there is little ditors into professional territories they This has opened a new Pandora’s Box, information yet about the effects of in- assumed were off limits to anyone with- where fatal social remedies and organi- creasing managerial flexibility and in- out research training. Not many re- zational efforts can easily result in more corporating other kinds of incentives for searchers were aware of the 1990 rather than less funding based on mis- changing the managerial culture. And report, Service Efforts and Accomplish- leading information or misinterpreta- the issues of what form performance in- ments Reporting: Its Time Has Come, tions of the data available. produced by the Government Account- formation should take for dissemination The GASB report suggests an ideal- purposes, to whom it should be distrib- ing Standards Board (GASB). The Board is a major player in developing guide- type chronology of steps which not only uted, how full utilization of this infor- equate auditors with evaluators but re- mation can be assured, remain thorny. lines and setting standards for the au- diting and accounting profession. duce both professionals to simple out- Clearly utilization is a major issue, since come measurement: none of these reform efforts will bear This report encouraged auditors to fruit without it. move into the measurement of out- n Policymakers authorize or re-autho- comes at the public agency level, re- rize programs and provide funding. garding the organization as a whole n Management collaborates with poli- Editor’s Note and/or its programs. What the GASB re- cymakers, customers, employees and port was all about was performance au- taxpayers to develop expected out- Additional GAO reports on perfor- diting, an entirely new specialization comes and measures of outcomes. mance management issues are: within the fiscal auditing world. This auditing function clearly overlaps with n Independent auditor performs finan- Program Performance Measures: Federal cial/legal compliance audits. Agency Collection and Use of Perfor- evaluation research in a period in which mance Data, May 1992. performance measurement is being n Internal/external auditors perform launched enthusiastically. Performance Budgeting: State Experiences program evaluations and verify re- and Implications for the Federal Govern- Driven by new economic scarcities porting of service efforts and accom- ment. February 1993. that were focusing attention on govern- plishments information. Improving Government: Measuring Perfor- ment size, waste, abuse and inefficien- n Policymakers identify, improve or mance and Acting on Proposals for cy, the performance measurement discontinue programs that have out- Change, June 1994. movement had begun in the late 1970s lived their usefulness, cease to make Managing for Results: State Experiences as the stress on greater public account- policy sense, can be run more effi- Provide Insights for Federal Management ability came into its own. This move- ciently, or have a lower priority than Reforms. December, 1994. ment was energized by stricter account- others. ability-oriented legislation at the federal Government Reform: Goal-Setting and Per- The growth of the performance mea- formance. March 1995. and state levels, and the other fiscal containment efforts in the 1980s. There surement movement in the 1980s and was a public groundswell for wanting 199Os has strengthened the welding of more detailed and sophisticated infor- the results of outcome studies of all mation on the results flowing from the kinds with major budgeting processes expenditure of tax dollars. and decisions. In this process the role of the auditing community has grown Performance auditing essentially in- stronger. And in a period of vastly re- volves the conducting of descriptive duced government resources, budget gross outcome evaluations by auditors deficit concerns, and anti-big-govern- untrained in social research principles ment sentiment, it looks far less expen- and methods. A performance audit ac- sive and more efficient to conduct cepts limited, general managerial expla- quick-turnaround performance audits nations of why and how the gross out- than longer descriptive outcome studies comes observed are occurring, but the by researchers and full blown net im- clear emphasis is on a small group of pact studies. But the risks of producing outcomes without attention to the rela- a narrow, biased information base, and tionship between interventions and out- overrelying on its use in allocating re- comes, or without much concern about sources are increasingly more serious. which outcomes can be attributed ex- clusively to a government agency or

Evaluation Forum n Issue 12 n Summer 1997 61 Evaluation Issues and Activities n The Performance Measurement Movement

nn In a frank and accurate assessment, Editor’s Note More Insights About Stern also notes the tendency for gov- ernment bureaucracies, in the midst of For a more detailed look at distinc- Performance the new accountability movement, to tions between performance auditing Auditing and focus only on certain categories of eval- and evaluation research, please see: uative and financial information. Such Chelimsky, E. “Comparing and Contrasting Evaluation Research data are being viewed as the most rele- Auditing and Evaluation,” in Evaluation vant information for measuring the ben- Review 9, 1985. efits of policies and programs to cus- Another perspective on the relation- tomers, stakeholders and society, when ship between evaluation research and in fact this information and the meth- auditing is provided by an interesting ods for collecting it are not always interview in the October 1995 issue of consistent with the scientific and pro- Evaluation. A new international evalua- fessional standards of evaluation re- tion journal published by Sage and edit- searchers. On the other hand, Stern sees ed by Elliot Stern of the Tavistock Insti- the nearly worldwide performance man- tute in London, its emphasis on “theory, agement/budgeting drive as opening up research and practice” has successfully new opportunities for evaluators, de- reduced the isolation of American eval- spite its sizable challenges. uators from applied researchers in other countries, broadening their professional Increasing Financial Control approaches and expanding their re- Pratley describes the considerable re- search repertoires. sponsibilities of the European Commis- This issue of Evaluation Forum re- sion’s Financial Controller in terms of a ports on an interview conducted by 150-person staff processing 360,000 Stern, an evaluation researcher, with transactions per year. This staff has the Alan Pratley, Financial Controller of the right and duty to monitor how all Euro- European Commission (EC) and Acting pean Commission funds are spent, one Director General of its Financial Control of the largest subfunds being the Struc- Directorate. The interview is a case tural Funds for national, regional and study of how auditors and evaluators local human service, community and see the world differently, talk past one economic development projects in the another, hold to their territorial views, EC’s member countries. However, only and are sliding into competition with since 1990 has the Financial Controller one another in a period in which “ac- been assigned the task of carrying out countability” is an all-important priority an ongoing, regularized program of in- of governments. ternal audits. In his introduction to the interview, With respect to the Structural Funds, Stern points to the growing emphasis on member states are required to develop both public sector evaluation and audit- adequate management and control sys- ing in all the European Union countries. tems. The EC audit division has the His comment is that government audit- right to access information for EC audit- ing “has been at the cutting edge of ing purposes, and the ability to require many contemporary regulatory and that member states perform their own public management initiatives.” He audits if problems are identified. The claims that as early as the mid-198Os, work of the EC audit office is overseen the evaluation research and auditing externally by the European Court of Au- professions were borrowing concepts ditors, which is in turn monitored by from each other, which has contributed the European Parliament. to the management concepts being for- Pratley refers to the new amendment mulated under increasing public pres- to the Financial Regulations (introduced sure to reform public management in- in 1991) as a significant change in EC ternationally. By the 1990s, Stern says, attitudes toward the auditing function. the U.S. government and many other This amendment required the develop- central governments were pushing hard ment of ‘’quantified objectives” and the for the institutionalization of perfor- monitoring of the extent to which they mance management and evaluation, were achieved in terms of efficiency and stronger linkage between evaluation and cost-effectiveness. Prior to this legislative auditing activities, and the integration revision, the Financial Controller had of evaluation results with budgeting de- concentrated on inputs and processes cisions. rather than outcomes and impacts. This reflected the pre-l990s experience of au- ditors internationally.

62 Evaluation Forum n Issue 12 n Summer 1997 Evaluation Issues and Activities n The Performance Measurement Movement

Financial Assessment Methods Stern deftly moves Pratley toward Evaluation Issues and Activities section the Controller’s current definition of of this issue on the European Commis- In Stern’s interview, Pratley is asked “evaluation,” by asking for a response sion guide for Monitoring Committees]. to explain the assessment strategies to what Stern has absorbed in his own The EC policy directorates, other than now used to audit progress against work: the auditing division, he believes, quantitative measures of performance. should be the sector collecting this kind So far you have identified Regarding the Structural Funds, which of information from managers, encour- evaluation with financial cover, among other things, innovative aging them to adjust their implementa- control and its concern with training programs, infrastructure devel- tion approaches based on such informa- sound management, the ef- opment projects and rural improvement tion. That does not mean, Pratley ficient use of resources, and projects, Pratley sees outcomes as being recognizes, that Financial Control achieving results. What is fairly easily measured. But Stern presses should ignore implementation issues in specific about evaluation? him further, based on his own experi- conducting performance audits. ence in evaluating Structural Funds pro- Pratley distinguishes between the Stern continues this line of thought, grams. He suggests that there is a ten- new EC model and the traditional however, asking why the goal of finan- dency to develop performance measures French system in which obedience to cial auditing should not be to determine only for the easy-to-quantify outcomes, rules is the main priority in financial the critical relationship between what while the more important outcomes ex- auditing. Now the emphasis is on pro- the results are and what appears to ex- pected of social initiatives are often in- viding evidence that you are doing what plain them, particularly in large-scale tangible, such as increased innovative- you proposed to do for the lowest possi- programs where one needs to know ness, coordination, and ble cost, or that you are applying cost- whether results represent a policy or a competitiveness. effectiveness principles to what you are program failure. But Pratley clearly doing and you have quantified your ob- Pratley’s response is sensitive to this leaves this kind of evaluation to other jectives and tracked results against problem. However, he sees great benefit divisions within EC or to external evalu- them. This, Pratley claims, is the mean- in requiring projects to describe more ators. precisely what population is involved, ing of “evaluation” within an auditing what areas are covered, and what their framework: making a judgment about activities and outcomes are — and de- goal achievement. To institutionalize Stakeholder-Oriented Auditing manding that these projects incorporate this concept of evaluation, Financial and Evaluation monitoring and evaluation strategies Control is developing “the performance Stern poses another interesting ques- within initial and ongoing project plan- audit.” tion, namely how the new EC financial ning. management framework will accommo- Stern also asks about potential collu- The Isolation of Evaluation date the emerging emphasis on stake- sion between the interests of the leader- Research from Performance holder-oriented evaluative approaches — ship of the EC and the internal auditing Auditing where those with a clear and significant division, related to the need to avoid investment in a program are brought As an evaluator, Stern cannot leave into evaluation planning, implementa- negative audit results in the context of the discussion at the point Pratley wish- scarce funding to continue the Structur- tion and utilization. Pratley’s response es to end it. He comments that evalua- is an intriguing non-answer. Substantial al Funds and other projects. Pratley tion researchers are as interested as views more rigorous evaluations as the concern across member countries of the new-age auditors in studying results. European Union about past misuse and purview of external experts. He is not But they are also concerned, he says, concerned that program managers may potential abuse of EC funds has led the about studying implementation process- EC to concentrate more strenuously on mask their true level of achievement in es that help explain results. an effort to preserve problematic pro- the way funds are used, according to grams. The EC financial audit division Objectives are not simply Pratley, and particularly how cost counters this possibility, he says, by em- the benchmark against effectively they are used. Focusing at- phasizing with managers that problems which results are judged — tention on fraud and abuse serves to encountered in programs are not neces- how they evolve and change send a message to stakeholders that the sarily due to poor management. This at- is also a focus of attention. EC is serious about an issue of interest to them. Pratley answers Stern’s ques- titude, he feels, reduces managerial anx- Pratley sees this broader and more iety and assures accurate reporting. tion about stakeholder involvement in holistic interest of evaluation research- the evaluative process by indicating that Also Pratley feels that the kind of ers satisfied by ongoing, if quite gener- this issue is satisfied by the narrow re- monitoring currently carried out by the al, feedback from managers about con- sponse to stakeholder worries about EC focuses appropriately on an adher- flicting objectives and implementation abuse. ence to the general framework and prin- problems, rather than from formal ef- ciples developed by the auditing divi- forts by internal auditors or outside ex- Later in the interview it becomes sion rather than on specific guidelines perts to determine why and how pro- clear that Pratley finds it important to — particularly the general commitment grams go astray or succeed. The distinguish between the formal role of to cost-effectiveness and ongoing moni- Monitoring Committees created in mem- evaluation research units within the pol- toring and evaluation. This general em- ber states to oversee Structural Funds icy directorates of the EC, and the finan- phasis, he says, gives managers suffi- programs, for example, would be more cial control division. The former, he cient leeway to produce trustworthy, appropriate bodies, he says, for identify- assumes, should conduct process, out- useful information. ing process (implementation) issues and come and impact evaluations when difficulties. [Please see the article in the needed to judge more comprehensive

Evaluation Forum n Issue 12 n Summer 1997 63 Evaluation Issues and Activities n The Performance Measurement Movement

nn longer-term efficiency and effectiveness. General of the Budget have jointly spon- Consequently these units could address sored a study of communities using a Comments on the a broader range of interventions, imple- team of nine experts in different fields mentation structures/functions, and re- — a more collaborative model — which Performance sults. Performance audits were not to go will provide information to all sectors of Measurement beyond a general judgment of efficiency the EC. He also wants a contact for his and cost-effectiveness à la money man- division developed within each internal Movement in agement practices, utilizing a set of bu- evaluation unit or a liaison in director- reaucratically-identified (rather than ates that do not yet have such units. Australia stakeholder-responsive) outcomes. Stern ultimately asks the most pene- Some Thoughts About the Sue Funnell gives us another interna- trating question for evaluators: to what Interview tional perspective on performance mea- extent are the worlds of evaluation re- In summary, what Stern evokes in surement in her paper about the Austra- searchers and performance auditors this interview between professionals lian approach to performance issues competitive, and even in potential con- with substantially different training and prepared for the International Panel on flict? In so asking, Stern acknowledges experience is an interplay that reveals Performance Measurement in Vancou- that the two professional territories are the perennial differences in conceptual- ver, Canada in November, 1995. Funnell conceivably at war in some European ization, interpretation, professional is a researcher with the firm Perfor- Union countries. Again, the Financial principles and methods, and scope of is- mance Improvement Limited in Huntleys Controller sidesteps this issue by claim- sues between those trained in social sci- Point, Australia. Her presentation drew ing that an active intolerance is required ence research and those trained in ac- not only interested evaluators in the Eu- toward managers in the EC directorates counting and financial management. ropean Union, North America and Asia and member countries who wish to Meanwhile, two new directions are evi- but also some inspector generals. The place their own organizational agendas dent in the European Union, as in the question period did not avoid debate ahead of obedience to the new perfor- U.S. and other countries: 1) the move- about the appropriateness of auditors mance audit framework. This says a ment of auditors into performance mea- moving into the assessment of perfor- great deal about how willing the finan- surement and interpretation, earlier the mance, but there was comfortable cial wing of the EC is to work with exclusive domain of evaluation re- agreement that the pressure to measure managers, internal evaluation units, and searchers, and 2) the close linking of performance was driven by economic external evaluators in a collaborative, performance auditing with budget deci- forces impinging on the governments stakeholder-sensitive way. sions which have far reaching effects on represented. However, Pratley does indicate that policies and the program strategies de- In most of the postindustrial coun- Financial Control and the Directorate signed to carry them out. tries, international economic competi- tion and associated fiscal pressures have led to an increased emphasis on man- agement, benchmarking, quality assur- ance, outcome measurement and contin- uous improvement processes. There has been a convergence across these nations even in the language used to describe these functions, tasks and processes. As with all these countries, the decentrali- zation of money flow and programs has placed greater emphasis on accountabil- ity at lower levels of government. Consequently the need for these lev- els to feed back information to central governments for resource allocation purposes has grown. Understandably, central governments have increased their demand for program outcome in- formation as they cope with a loss of control over program design and imple- mentation. In the new social policy cli- mate, countries have sought to balance concern about the value of social pro- grams to their recipients with increased cost efficiency as organizations. This has led to linking performance measure- ment to budgeting.

64 Evaluation Forum n Issue 12 n Summer 1997 Evaluation Issues and Activities n The Performance Measurement Movement

Australia is a useful case example. n Most information about cause-effect used and with what effect. Rather than Funnell reports that performance mea- relationships is based on monitoring simply measuring gross outcomes, the surement has taken several forms in data rather than on more rigorous focus is on net impact, but adequate Australia, remarkably similar to trends evaluations. methodologies for determining the latter in the U.S. and the European Union: are not being used due to the lack of Customer Satisfaction, focusing on funds and research expertise. Benchmarking, focusing on out- “customer service guarantees” or the comes or best practices. The main is- quality of the exchange between service Funding for evaluation has been sub- sues have been the comparability of the deliverers and service receivers. Funnell ject to the same emphasis on cost effi- data used to develop benchmarks, the emphasizes several issues. One is what ciency as other government functions, potential for the misuse of data, and the qualities of the above exchange are best so there are only minimal funds for controversy over whether benchmarking to measure. Some facets of this relation- evaluation and these generally go to in- should focus more on end results based ship may not be directly related to the ternal evaluation units. In most cases, on quantitative data than on informa- outcomes of service delivery. Also, at- agencies conduct self-evaluations with tion about the nature of implementation tention to client satisfaction may lead to existing staff, although some evaluation processes. There is concern that govern- a neglect of the needs of families, other and performance measurement training ments will prefer quantitative bench- groups, and the community. Further- is provided despite central government marking exclusively, which Funnell be- more, some of the measures of custom- ambivalence about the expense. Select- lieves could raise confidentiality issues er satisfaction may be so superficial that ed agencies are able to use third-party and be misleading as a criterion for they lose their utility in getting at more independent evaluators. budgeting decisions. critical customer satisfaction variables. A more general issue identified in the Total Quality Management, focusing Continuous Quality Improvement, Funnell paper is the fact that formal on the measurement of the quality of focusing on information that can im- support for more expert program evalu- management processes. The key issue, prove practice — both processes and ations is decreasing at the central gov- according to Funnell, is whether private their impact. Australia is currently using ernment level in Australia, which paral- sector concepts can be applied appro- a hierarchy of six stages of organization- lels the trend elsewhere. The fear is that priately to public sector management, al development as a framework for the critical linkages between inputs, which must deal with more ambiguous studying continuous improvement in processes, outputs and impacts will be definitions of efficiency and effective- agencies and programs. A concern is neglected. Consequently the outcomes ness. that agencies must be motivated to en- being measured may be assumed to tell gage in activities consistent with the the government what the exclusive re- Quality Assurance, focusing on the continuous improvement concept, and sults of a given program’s interventions assessment of the quality of organiza- this is difficult in the context of repeat- are, when the information simply does tional processes and procedures, often ed budget cuts and the devaluation of not support such conclusions. using a set of quality standards. Impor- the role of government and its accom- tant issues are whether the use of stan- Excessive emphasis on measurable plishments — a trend occurring in all dards is appropriate, how relevant these outputs with minimal attention to net the postindustrial countries. standards are, and whether they actual- impacts is an obvious danger cited by ly improve quality. Clearly there are Australia not only emphasizes the Funnell. Also there are gaping holes, problems with these assessment strate- forms of performance measurement out- she says, in the information collected by gies, mainly in terms of objectivity, cov- lined here, but has a formal mandate agencies regarding the size and charac- erage of important factors, and research and timetable: teristics of target groups, the nature of expertise. target group needs, and the extent and n Assessments of the need for certain features of service delivery. To address The Australian Inspector-General programs must be made. these concerns, a number of govern- conducts periodic reviews of the quality n Annual performance reports are re- ment-sponsored performance measure- of performance information. A review in quired for all programs. ment activities are underway, particular- 1993 provided these conclusions about ly the collection of performance indi- the information base: n Evaluations of all programs are to be cators in nine areas to compare their conducted on a cycle of 3-10 years, to n The quality of program management ability to measure efficiency and effec- determine their appropriateness, effi- tends not to be related to program tiveness — police, courts, corrections, ciency and effectiveness. outcomes. schools, vocational education, hospitals, n The central government publishes a aged/disabled services, family/child n The measurement of management register of completed evaluations welfare, and public housing. And mea- quality is often based on unstated or twice a year. For each evaluation, the surable performance standards are be- questionable assumptions about the register summarizes its purpose, ing established which reflect the level link between program strategies, re- methodology, and major findings. and quality of services. sources, outcomes and performance. Evaluations are expected to be an on- n The quality of information about going management responsibility, and management does not provide a are an integral part of budgeting and good basis for assessing management planning. They are management-driven, or program performance. emphasizing how resources have been

Evaluation Forum n Issue 12 n Summer 1997 65 Evaluation Issues and Activities n The Performance Measurement Movement nn of budget expenditures could be ex- Response from Attenders at the Comments on the plained based on outcome data. Conse- Performance Measurement Panel quently the Ministry of Finance required Performance that annual budgets provide evidence of The international conference was Measurement the contribution outcome data on pro- jointly sponsored by a number of Euro- grams had made to an understanding of pean and Asian evaluation research as- Movement in the particular budget items. Despite this sociations, yet a large contingent of au- more specific directive, Leeuw’s conclu- ditors attended. This was clearly related Netherlands sion from analyzing central budget in- to the movement of auditors into the formation is that government agencies performance audit field, which is gain- were continuing to resist the use of per- ing credibility world-wide and begin- The interest in performance mea- formance measures in interpreting ex- ning to crowd evaluation research as the surement is now an international obses- penses. In addition, the central govern- strategy of choice in determining gov- sion, fueled by common economic prob- ment had been slow to institutionalize ernment and program performance. lems throughout a global market. It is policy and program evaluations, even The response to Leeuw’s concerns also a correlate of decentralization and though academics have urged that it do about the downside of performance privatization, which have transferred so for two decades. measurement were well received.The accountability to lower levels of govern- central tendency in the audience was to ment and private sector service deliver- worry about the rush to quick judg- ers. The Netherlands is a case in point. The “Performance Paradox” and Its Unintended Consequences ments of government operations and programs via a narrow set of perfom- In this brief article we review a con- Leeuw sees the “proliferation and ance measures and standards, without ference paper by Frans L. Leeuw pre- noncorrelation” of performance mea- sufficient attention to broader theoreti- pared for the Panel on International sures as a new paradox to be addressed. cal measurement and methodological is- Trends in Performance Measurement at He points out that performance mea- sues in determining efficiency and the International Evaluation Conference sures tend to be only weakly correlated. effectiveness.The auditors present were in Vancouver, Canada in November He also feels there are too many of not so much concerned about these is- 1995. Leeuw is professor of sociology at them. Beyond that, he claims the in- sues, but rather about the need for addi- the University of Utrecht and serves on creasing use of performance measures tional staff trained in performance au- the Netherlands Court of Audit in the to monitor programs is leading to the diting and more funding to add such Hague. growth of “second order performance staff. But the strong trend toward study- assessment” by external auditors and fi- ing the results of government activity Accountability Strategies nancial analysts rather than analyses by was everywhere evident. The central government of the Neth- evaluation researchers. And he suggests erlands is guided fiscally by a Budget that performance measurement can and Accounting Act which requires have other negative effects: government ministers to carry out sev- Editor’s Note n It can inhibit innovation and lead to eral kinds of periodic assessments: organizational paralysis. financial management audits; perfor- For additional references, please see: mance measurements; and policy and n It can lead to an excessive reliance program evaluations. The Netherlands on performance standards. Leeuw, F. L. And P. J. Rozendal. “Policy Eval- Court of Audit is an independent body uation and the Netherlands Government n It can focus attention too exclusively that performs assessments for the cen- Scope, Utilization and Organizational on the quantifiable aspects of perfor- Learning,” in Can Governments Learn? tral government. Unlike the United mance. Kingdom, there are no comparable local edited by F. L. Leeuw. New Brunswick, United Kingdom: Transaction Publishers, government offices. 1994. Leeuw’s analysis focuses singularly Meyer, K. and K. O’ Shaugnessy. “Organiza- on central government performance Figure 1 n Leeuw’s List of Policy tional Design and the Performance Para- measurement and evaluation regarding Interventions dox” in Explorations in Economic Sociol- the use of the following array of inter- ogy edited by R. Swedberg. New York, ventions (see Figure 1). Leeuw’s thesis Subsidies NY: Russell Sage Foundation, 1993. is that information on the efficiency and Quangos Smith, P. “On the Unintended Consequenc- es of Publishing Performance Data in the effectiveness of these policy tools is mini- Public information campaigns mal. Public Sector,” in International Journal of Government-sponsored enterprises Public Administration 18, 1995. In 1992, the central government de- cided to require more information on Contracts outputs and effects, but a subsequent Permits survey indicated that only fifty percent Inspectorates Legislation Loans and guarantees Levies Fees and charges

66 Evaluation Forum n Issue 12 n Summer 1997 Evaluation Issues and Activities n The Performance Measurement Movement

nn hand, the use of a less formal perfor- mained weak. Managerial energy was Improving mance system in the Massachusetts ET still narrowly focused on reporting the Choice program, and in the GAIN pro- number and type of clients participating Performance gram in Riverside, California — both in JOBS activities. Bartik points out that Systems in employment and training programs — the federal government literally did not indicated that clear-cut performance know who got jobs or left the welfare Supporting Program goals and incentives did improve goal system on an annual basis. achievement. Goals Leaning towards the GAO’s position, Bartik’s thesis is that “it is feasible to The Proposition That develop good indicators of performance The first systematic performance Performance Standards for a particular welfare-to-work pro- measurement system for social pro- Systems “Work” gram, office or contractor.” He believes grams was the performance standards/ The potential positive effects of per- that performance indicators can moti- incentive system developed for the fed- formance standards are the focus of a vate key actors to be more effective in eral Job Training Partnership Act (JTPA) June, 1995 monograph published by W. achieving a program’s true goals, and programs authorized in 1982. Legislat- E. Upjohn Institute for Employment Re- can provide information about “best ing the use of standards and a mecha- search, Using Performance Indicators to practices.” To accomplish this, he pro- nism for assuring their application was Improve the Effectiveness of Welfare-to- poses that performance measurement driven in large part by perceived weak- Work Programs. The author, economist systems must be simple, timely, and nesses in the previous Comprehensive Timothy J. Bartik, details the history control for factors other than the pro- Employment and Training Administra- summarized at the beginning of this ar- gram that influence whether or not wel- tion (CETA) program. Because of the de- ticle, placing emphasis on the use of fare clients achieve the goals intended centralized and privatized nature of performance management systems for for the program. JTPA, the program involves an applica- welfare reforms. Although the current Key program goals, he proposes, are tion of standards at the federal level for national welfare program, JOBS, was to improved self-esteem, better job skills, states, at the state level for regional Ser- be reinforced by a performance system employment, earnings gains, and re- vice Delivery Areas (SDAs), and at the similar to that used in JTPA, Bartik ex- duced welfare dependence. Simple pre- local level for service providers. States plains that the federal government post- post outcome evaluations will not sur- are allowed to adjust standards for sub- poned recommending performance face such improvements, he says. Ex- state areas based on measures of certain measures for JOBS until October 1996, perimental net impact studies can do variables felt to be major factors outside and performance standards until Octo- so, but they are often costly and time- the control of State Job Training Coordi- ber 1998. This is long past the presiden- consuming, and are not likely to be nating Councils that can influence their tial election which will surely direct the conducted on a regular or widespread performance. Up to 5% of program form future welfare reform initiatives basis. Also, the latter raise ethical ques- funds flowing to states are to be used to will take. The manifest reason for the tions about withholding interventions reward SDAs for complying with or ex- delay, he says, was sensitivity to the from eligible clients. The advantage of ceeding standards, or to provide under- possibility that outcome-based perfor- performance indicators and standards, performing SDAs with technical assis- mance measures and standards have the Bartik says, is that their use does not re- tance intended to raise their level of potential for being misleading indicators quire such rigorous evaluations. The desired outcomes vis-a-vis standards. of program goal achievement. downside, however, is that they may A number of studies have examined Bartik refers the reader to the Octo- distort the intended value of a program. the effects of using such a system. In a ber, 1994 report of the U.S. Department But Bartik sees light at the end of that 1985 study for the U.S. Department of of Health and Human Services which tunnel in the form of “better perfor- Labor, Katharine Dickinson and Richard summarizes research by the Manpower mance indicators” and “careful use.” West found that states emphasizing the Demonstration Research Corporation on goal of exceeding standards spent more state welfare reforms. Differences in av- Better Indicators and More funds on each terminee. However, erage outcomes across these initiatives Appropriate Utilization spending per client entering employ- often showed no correlation with goal Bartik suggests a number of cautions ment was no higher. A 1993 study by a achievement. Within states, there was in developing more effective indicators doctoral student at Stanford University, often variation across different counties. for performance management systems: Michael Cragg, found that the areas re- Sometimes the correlation between ceiving the greatest rewards were enroll- standards and goal achievement was ac- n Performance indicators must be ing individuals with more work experi- tually negative. And program outcomes linked to information on the strate- ence. However, the more experienced within counties were often greater for gies of particular program organiza- individuals had a lower gain in earn- the less disadvantaged. tions and staffs so that we know why ings. Additional studies suggested that a particular program component is Meanwhile, in a 1995 study the U.S. SDAs were resisting enrolling their quo- successful. [This sounds like a rec- General Accounting Office (GAO) con- ta of welfare clients due to the fear of ommendation to explore all of these tended that the delay in developing failing to “perform” adequately. aspects of programs: organizational standards for JOBS was damaging this elements, implementation strategies, These and other studies suggested national program. Managers were not program interventions and out- that performance standards might not focusing enough attention on the em- comes.] be enhancing the achievement of the ployment of clients, a major goal of program’s intended goals. On the other JOBS. Linkages with employers re- n Indicators need to be timely and un- derstandable.

Evaluation Forum n Issue 12 n Summer 1997 67 Evaluation Issues and Activities n The Performance Measurement Movement

n It must be easier to increase perfor- performance standards raises important mum fairness criterion”. The perceived mance by achieving a program’s in- issues of feasibility, fairness and afford- fairness of a performance management tended goals than by selecting clients ability. system may depend on whether the sys- who will help managers meet perfor- tem affords flexibility regarding influ- mance standards. Guidelines for Developing ences not captured by quantitative ad- justments, such as reasons certain areas n Program managers must be motivat- Performance Indicators find it difficult to meet a particular stan- ed to use performance measures in al- Bartik concludes his monograph with dard. Fairness is also related to the size locating resources. suggested guidelines for developing in- of rewards and sanctions and how hard dicators or measures for performance n Indicators should be simpler and less standards are to meet. accurate as one gets closer to the indi- systems in welfare-to-work programs. 3. “Creaming” may be necessary and vidual staff level, and more statisti- They have significant implications for a good, if it is the right kind. cally sophisticated at higher levels of host of human resource efforts: the welfare system. [The reason Bar- 1. Agreement is needed on a few easy- Bartik says that making a selection tik gives is that “it is easier at the to-understand performance measures among welfare clients in providing ser- staff level to substitute personal in- that are frequently measured and vices sometimes supports the purpose teraction and judgment for quantita- which focus staff on the program’s in- of programs. For example, creaming that tive performance indicators” and tended goals. increases job development may have a “more difficult to consistently inter- positive value, since job development pret more complex performance indi- For example, the author of the report usually requires some level of selectivi- cators”.] on the ET Choices program concluded ty. One reason for the success of effec- that its apparent success was dependent tive job development, Bartik believes, is What Bartik says is new is a range of on having a simple goal consistent with that employer information on job appli- performance indicators, from quite sim- its mission. The same conclusion held cants and new hires is not very good, ple, timely, easily quantifiable measures for the JOBS program in Maryland, leading to costly employee turnover. that are admittedly “rough proxies” for which focused on three simple goals: Consequently, employers often hire the program goals, to “more sophisticated, the number of participants, the number employees, friends or contacts of the accurate and complex approximations” of participants leaving the program due employer, making it more difficult for of these goals. States, he says, have de- to increased earnings, and the job reten- those without a job development advo- veloped a variety of performance mea- tion rate for those who obtained employ- cate to be hired. In addition, new hires sures, and many have established incen- ment. obtained in this way are more subject to tive/sanction systems to encourage their the prejudice of employers. Screening use. Simple systems, Bartik insists, have a more immediate effect in motivating by programs to identify and market the The emphasis on a range of indica- staff than more complex ones. But these more productive members of caseloads tors is reinforced, according to Bartik, simplified systems are not without prob- can lead to better goal achievement. by the 1995 GAO study and a report by lems. Simple goals, he says, may con- Bartik’s qualification to this conclu- the American Public Welfare Associa- tain inconsistencies or actually conflict. sion is that strong enforcement of laws tion. States leading in the performance This problem is illustrated by the five against discrimination and careful mon- system movement have used measures categories of JOBS goals: job placement, itoring of job developers’ attitudes and of an array of outcomes, including ini- placement quality, reduction of welfare practices are necessary in order to pre- tial, intermediate, short-term (at termi- payments or getting off welfare, job and vent inappropriate screening. He agrees nation) and longer-term outcomes. They welfare reduction retention, and in- with Daniel Friedlander of MDRC that a have also negotiated goals with local of- creased job skills. If all these goals are “triage” approach to selecting program fices, and adjusted them based on un- to be measured, a complex system is in- participants may increase goal achieve- employment levels and the composition evitable, even though vigorously pursu- ment, where caseloads are classified of caseloads. Within different state ori- ing only a few goals may lead programs into subgroups based on the extent of entations, meeting and exceeding stan- in the wrong direction. their prior earnings and welfare history. dards have various consequences — In a 1988 study of employment and from no impact on funding or program 2.A performance indicator system training programs for welfare clients, operation, to more acceptable perfor- does not need to be totally fair to be a Friedlander found that there were posi- mance levels, to extremely strong re- useful motivator. tive earnings impacts for the “mid-de- wards and penalties. In this sense, Since it is impossible, in Bartik’s pendency tier” but none for subgroups states clearly have different objectives view, to measure perfectly or easily all in the “most dependent tier”. The least for their performance management sys- the major social goals of a program, disadvantaged clients tended to increase tems. It is therefore difficult to identify performance incentives and sanctions their earnings without assistance from any one system as being more useful may not always reward or punish per- programs. The message was that the than another. formers appropriately. The utility of a main goal of these programs (earnings The privatization of program opera- performance measure, he says, should gains) would be benefited by targeting tion and service delivery is another sig- be defined in terms not only of its accu- welfare recipients with “average difficul- nificant variant, where private firms are racy in representing the program’s goals ty”. paid based in whole or in part on their but also its ability to resist managerial The wrong kind of creaming clearly performance using a set of criteria. Bar- manipulation associated with meeting involves targeting clients whose out- tik suggests that privatization combined performance standards or targets. Stan- comes will best serve the purpose of with large incentives to meet certain dards clearly need to “meet some mini-

68 Evaluation Forum n Issue 12 n Summer 1997 Evaluation Issues and Activities n The Performance Measurement Movement

meeting performance standards but may velop than those for welfare reduction. programs had greater long-run success not represent the achievement of pro- And more adjustments to the indicator than other reforms. Bartik concludes gram goals. The targeting of certain cli- of welfare reduction are needed. A pro- that we do not yet know, and must do ents, Bartik suggests, must therefore be gram can look successful simply be- more research to determine, how to reli- determined externally, rather than by cause many recipients who have in- ably increase the long-run earnings of staff whose behavior is judged against creased their earnings are still individuals who would otherwise have performance standards. The main basis dependent on the welfare system. been long-term welfare recipients. for judging compliance with standards 6. It is more important to adjust for cli- must be the entire caseload, Bartik says, ent characteristics than to extend the Performance Measurement or an entire segment of the caseload, as length of the measurement period. Models suggested by a 1995 U.S. Health and Human Services report to the Congress. Extending measurement points be- Bartik’s concluding section suggests yond the thirteen-week follow-up period a number of performance models: 1) for 4. Data on desired program outcomes helps, Bartik suggests, but it does not state monitoring of local office perfor- for welfare clients should be adjusted seem to make the correlations between mance, 2) for state analysis of those through regression models so that the performance indicators and the true program approaches that appear to have adjusted numbers are positively corre- achievement of program goals any the greatest long-term effectiveness, and lated with the true goals of the pro- stronger. Bartik recommends that in ad- 3) for local office monitoring of staff gram. justing performance indicators for the and contractors. This monitoring hierar- Daniel Friedlander and Jeffrey Zor- unique characteristics of each welfare chy involves all levels of a decentralized nitsky, of Abt Associates, examined the office, researchers should consider these social program environment within a extent of the positive correlation be- factors: 1 ) the prior earnings history, single but flexible performance mea- tween adjusted outcome data and pro- prior welfare history, race, marital surement system. Clearly it requires a gram goals. The study suggested that status and gender of household head, substantial change in the managerial when individual outcomes are adjusted education and age of the client, and culture of social programs. for the individual’s characteristics and 2) measures that reflect both wages the characteristics of the local economy, and job availability. they are often significantly correlated 7. Using performance systems to im- Editor’s Note with the goals of the program for that prove the achievement of program individual. This kind of adjustment is Some of the studies referred to in goals. an incentive to avoid creaming. It also the Bartik monograph are these: increases the perceived fairness of the A major objective of performance American Public Welfare Association JOBS: performance system. And it is critical to systems is the development of indica- Measuring Client Success. Washington, replicating the program. tors that can be used in the short-run as D.C.: Institute for Family Self-Sufficiency, indicators of long-term success, using Friedlander found that even very 1994. the short run measures for program primitive adjustments to outcome data Anderson, K. R. Burkhauser, and J. Ray- management purposes. Using long-term can yield significant positive correla- mond. “The Effect of Creaming on Place- measures as a basis for incentives and tions. Zornitsky found these kinds of ment Rates Under the Job Training Part- sanctions would be ineffective for man- adjustments for individuals to be nership Act,” in Industrial and Labor agers because of the delay in determin- Relations Review, 46, 1993. significantly positively correlated with ing compliance. estimated earnings gains. Entered em- Behn, R. Leadership Counts: Lessons for ployment rate, placement wage, em- Also, the use of such measures, such Public Managers from the Massachu- ployment status and average weekly as more education, placement in better setts Welfare, Training and Employment Program. Cambridge, MA: Harvard Uni- earnings at three and six months, and jobs, or targeting chronic welfare recipi- versity Press, estimated reductions in average welfare ents is not always supported by the payments were also correlated. data. Some programs emphasizing edu- Friedlander, D. Subgroup Impacts and Per- cational interventions, for example, are formance Indicators for Selected Welfare 5. Performance measures and stan- Employment Programs, New York, NY: successful and some not. The Minority dards need to be as directly related to MDRC, 1988. Female Single Parent program in San program goals as possible. Jose, and the New Chance program, are Friedlander, D. and G. Burtless. Five Years In the Friedlander and Zornitsky examples of programs emphasizing edu- After: The Long-Term Effects of Welfare- to-Work Programs. New York, NY: Rus- studies, performance measures used to cation and skill development that have sell Sage Foundation, 1994. represent increased earnings and em- been judged less than successful. ployment were more strongly correlated Nightingale, D. et al. Evaluation of the Mas- It is difficult, Bartik says, to deter- with programs’ earnings goals than sachusetts Employment and Training mine from the data whether individuals Program, Washington, D.C.: The Urban were welfare-reduction measures. Wel- placed in better jobs do better later be- Institute, 1991. fare reduction performance measures cause of the placement, or whether in- were more strongly correlated with the Riccio, J. D. Friedlander and S. Freedman. dividuals with better skills get better effect of welfare reductions on programs GAIN: Benefits, Costs, and Three-Year jobs now and later. In the Friedlander Impacts of a Welfare-to-Work Program. than were earnings and employment study of the Baltimore program there New York, NY: MDRC, 1994. performance measures. Bartik’s conclu- was no evidence to indicate that welfare sion is that performance measures for Zornitsky, J. and M. Rubin. Establishing a recipients obtained higher paying jobs earnings goals are probably easier to de- Performance Management System for than in other programs, even though the Targeted Welfare Programs. Washington, D.C.: National Commission for Employ- ment Policy.

Evaluation Forum n Issue 12 n Summer 1997 69 Evaluation Issues and Activities n The Performance Measurement Movement

Other W E. Upjohn Institute mono- thank the authors of vari- graphs by Timothy J. Bartik that may be The Potential Effects of ous chapters for their of interest: useful review of our arti- Performance Standards cle and their suggestions The Effects of Metropolitan Job Growth on the Size Distribution of Family Income, Systems for improving it. 1991. Federal Policy Towards State and Local Why We Need to Know the Effects of Economic Development in the 1990s, with the assistance of Using Performance Standards 1993. Ernst W. Stromsdorfer The performance measure- The Effects of Local Labor Demand on Indi- Department of Economics ment movement has taken vidual Labor Market Outcomes for Differ- Washington State Universi- hold in all the postindus- ent Demographic Groups and the Poor, ty trial nations in the 1993. 1990s. There is a direct Economic Development Strategies, 1995. relationship between the In this article we re- Can Economic Development Programs Be growth of central govern- Evaluated, 1995 (With R. Bingham) view an early draft of a ments over the past few significant book to be decades, increasingly wor- published soon by the W. risome central government E. Upjohn Institute for deficits, the decentrali- Employment Research, Per- zation of social initia- formance Standards in a tives, and the need for Government Bureaucracy, central governments to edited by James J. Heck- compensate for the loss of man. A well known labor control over social pro- economist at the Universi- grams receiving central ty of Chicago, Heckman has government money by focus- contributed important in- ing accountability efforts formation in the draft on their results — i.e., manuscript to those who their performance. Perfor- have watched fascinated as mance standards have be- the performance measure- come a major tool in this ment movement has unfold- new effort to be account- ed, curious and anxious able in the midst of sub- about its longer-term im- stantial government re- plications. structuring and Heckman is Director of downsizing. the Center for the Evalua- Little conclusive re- tion of Social Programs in search has been conducted the Irving B. Harris on the effect of perfor- School at the University, mance standards on deci- and is Henry Schultz Dis- sionmaking. This is partly tinguished Service Profes- due to the current politi- sor in the University of cal and bureaucratic popu- Chicago’s Department of larity of the strategy. Economics. The manuscript However, it is important has the strong imprint of that we give performance Heckman’s thinking and re- measurement attention pri- cent research by way of or to applying performance several chapters co-au- standards to increasing thored with colleagues at numbers of national pro- the Center, and through grams. The most critical the general framework he reason the performance- developed for integrating oriented movement should the manuscript. be studied for its impact The editor of Evaluation is the need to understand Forum and the guest re- whether, to what extent, viewer, Ernst Stromsdorf- and in what ways perfor- er, would like to express mance standards, rein- our appreciation to Allan forced by financial re- Hunt, W. E. Upjohn Insti- wards and corrective tute, for permitting us to sanctions, may redirect write a review article the attitudes, decisions based on a draft copy of and actions of those set- the manuscript. We also

70 Evaluation Forum n Issue 12 n Summer 1997 Evaluation Issues and Activities n The Performance Measurement Movement

ting program policy and selves. Anticipating that Heckman and his col- managing programs away these criteria may be leagues focus their analy- from a program’s intended quite important, Heckman ses on the behavioral im- goals, objectives, imple- proposes that formal eli- plications of performance mentation strategy, and gibility and managerial standard systems for the desired outcomes. We need acceptability should be JTPA program, a reworked to determine the value or treated as distinctly dif- and expanded model of detriment in any redirec- ferent participant selec- which has been proposed by tion that is occurring. tion criteria, whose ef- the Clinton Administra- fects need to be carefully tion. It is an appropriate There have been persis- identified and analyzed. model beyond its political tent worries about perfor- saliency. The program’s mance-driven program de- legislative design incor- signs in terms of their Assumptions Underlying Performance porates performance stan- potential for distorting Measurement Systems dards and incentives for or reformulating policies Heckman’s explanation meeting them — JTPA is the and procedures for select- for the growing interest first national program to ing program participants — in performance measurement make these elements a re- that is, redirecting the is the belief that govern- quirement. selection process away ment is neither suffi- from the program’s selec- ciently efficient nor ap- In addition to using a tion objectives. In em- propriately sensitive to major ongoing national ployment and training pro- its customers’ needs and program as the book’s case grams, the temptation to concerns, resulting in study, the team of ana- “cream” the eligible popu- pressure for greater lysts benefited from the lation of clients for the privatization and govern- extensive data available higher-skilled individu- ment reform. The premise on JTPA through the exper- als, whose performance is rationalizing privatiza- imental National JTPA most likely to assist man- tion, he says, is that Study completed in 1995. agers in meeting perfor- market competition tends These data included infor- mance standards, has been to lower costs and in- mation for each of sixteen a major concern of poli- crease sensitivity to cus- study sites concerning 1) cymakers and evaluators. tomers. Consequently, the the structure of incen- tives, 2) changes in these For example, a 1991 U.S. argument goes, the segment incentives over the period General Accounting Office of government that remains of the study, and 3) the report claimed that per- unprivatized needs to bor- nature of the decisionmak- formance standards had re- row heavily from market ing process and its out- sulted in serious inequi- concepts, such as market- comes. ties, reducing the access oriented incentives that of women and minorities to can push government to- The series of re-exami- the program, to training wards greater efficiency nations of the experimen- within the program, and to and responsiveness. tal data which are dis- the more productive train- Heckman views a system cussed in the book ing alternatives. The re- of performance standards, represent a test of an im- searchers found evidence and a financial reward portant cause-effect rela- that only the “most em- system supporting the tionship: do performance ployable” disadvantaged achievement of these stan- standards and incentive workers were being served. dards, as the most impor- systems change a program’s They attributed this re- tant aspects of this intended design —its in- sult to the strong influ- transplantation of private tended participants, in- ence of incentive payments sector ideas into govern- terventions, implementa- and the prestige accorded ment activity. The goal of tion strategy, and “high performance.” this version of government outcomes? However, Heckman cites reform is to simulate pri- The major research ques- other possible sources of vate sector incentives and tion for Heckman and his program bias, such as man- competitiveness in a non- research group, across the agement decisions favoring market environment, ener- series of studies, was how certain client character- gizing government to be market-simulating systems istics that are unrelated more cost-effective and affect the behavior of bu- to a person’s suitability satisfying to its custom- reaucratic administrations for training or a concern ers. and managers. More specif- over equity issues, and ically, the search for an- the personal choices of The Test Case swers to the following eligible clients them- questions guided the anal-

Evaluation Forum n Issue 12 n Summer 1997 71 Evaluation Issues and Activities n The Performance Measurement Movement

yses: assisted Heckman and his strated considerable inno- fellow evaluators in iso- vation in manipulating How do performance stan- n lating the role played by various components and dards operate to exclude bureaucratic decisionmak- features of the program to or include persons with ing. Furthermore, the ex- boost measured perfor- different characteris- perimental study contained mance. These practices in- tics? information on over 20,000 cluded enrolling persons n Do performance standards individuals, permitting an only after they obtained a affect the type of ser- analysis of the effects of job, choosing short-term vices offered? events in the recent em- training as a way to in- ployment history of these crease the probability of Do the outcome measures n individuals on their prob- sufficient placements, used in developing the ability of being selected holding less employable performance standards into JTPA. The National individuals in job clubs, predict the long-term JTPA Study also provided and terminating those do- gains in participants’ data on the JTPA-eligible ing poorly in training earnings that are the nonparticipants in the slots at a point advanta- primary goal of JTPA? geographical areas of the geous to meeting perfor- n How successful are per- sixteen sites, and afford- mance standards. Such formance standards in ed rich data about varia- practices are clearly ra- achieving their own tions in the applications tional and efficient in stated goals? of performance standards meeting performance stan- in these sites. dards, but contribute lit- What general lessons can n tle in raising partici- be learned from a study pants’ earnings or of the effects of varia- Major Themes increasing service equity. tion in the rules gov- Heckman’s chapter, “Les- erning JTPA employment sons from the JTPA Incen- The short-term perfor- and training centers? tive System,” which ends mance measures used in setting standards were These questions did not the manuscript, summarized weakly (and often nega- emerge in a vacuum, even the major insights emerg- tively) related to though the literature on ing from the series of longer-term effects on the impact of performance studies. These are the employment and earn- standards has been meager. significant “lessons” ings. Heckman sees JTPA’s In addition to the GAO Heckman distilled from the intended goals as long- study, the 1988 SRI Inter- analyses: term effects, not the national study of the ef- Performance standards short-term requirements in fects of performance stan- exerted a powerful in- performance standards. dards on participants’ fluence on management Neither the original set choice to enter the pro- and staff behavior. Al- of standards, to be ap- gram, their choice of though JTPA financial in- plied at a participant’s training options, and the centives are comparatively termination from the pro- cost of training suggested low and their benefits gram, nor the new set man- that an emphasis on per- general, they have a dated in the JTPA amend- formance standards de- strong influence. This is ments, to be applied 13 creased services to the due to their value in in- weeks following termina- “hard-to-serve.” A study creasing the professional tion, were highly corre- of Tennessee’s JTPA client stature and prestige of lated with JTPA’s intended population in 1993 re- bureaucrats, and the le- impacts. Consequently, the vealed a disproportionate verage that awarded dol- influence performance number of individuals in lars create with the human standards clearly had on the participant population services community in managerial behavior did who had a high probability which administrators and not appear also to promote for successful job place- managers wish to be credi- long-term wage gains or ment, one of JTPA’s key ble players. positive employment out- performance standards. Even small rewards are comes for participants. However, these studies clearly significant. JTPA JTPA performance stan- did not benefit from de- agencies were making their dards did not result in tailed longitudinal micro- placement, enrollment and any significant “cream- level information on stag- termination decisions con- ing.” Although previous es in the selection and sistent with the need to research suggested the participation process. meet performance stan- possibility of creaming, This information greatly dards. Management and for most demographic staff practices demon-

72 Evaluation Forum n Issue 12 n Summer 1997 Evaluation Issues and Activities n The Performance Measurement Movement

groups the earnings ef- extent to which diverse greater equity in resource fects were uniform over a demographic groups dif- distribution, and 3) fair- range of skill levels. fered in their proportion ness — in that the hard-to- Only in the lowest skill in the participant popula- serve may be under-repre- category was there evi- tion at various steps in sented in the participant dence of smaller impacts. the selection process, 2) group due to discriminato- There was a trade-off be- determine the demographic ry practices. tween serving the most characteristics of the el- Heckman alone examines disadvantaged and using igible JTPA population in the issue of whether resources most efficient- the four sites who had short-run performance ly, but only a very modest heard of the program, were goals, such as those used one given JTPA’s modest aware they were eligible, in the JTPA performance impacts. Administrators’ who were applicants previ- standards system, predict scope of autonomy and dis- ously, or who had been long-run impacts of the cretion played a role in participants at some prior kind desired in the JTPA explaining differences in time, and 3) provide an legislation. If these client participation rates empirical basis for de- short-term outcomes are but not the predominant signing econometric selec- not positively correlated one. Information about the tion estimators for a sub- with longer-term goals, program and its service sequent study of the the assumption is tenable options, and the personal selection process. that these standards have choices of clients played Pascal Courty and Gerald guided bureaucratic behav- a more important part in Marschke of the University ior away from legislative explaining the disparities of Chicago Economics De- intent. in participation rates for partment and the Center, different groups. The more Courty and Marschke fol- study the organization of employable and more easily low Heckman’s analysis the “JTPA bureaucracy,” in placed applicants were with a study of actions by particular its unique gov- more likely to be encour- JTPA bureaucrats which may ernance structure. They aged to enroll, but these be motivated primarily by examine how the JTPA in- were the individuals most performance awards. They centive contracts associ- likely to be more aware of focus their attention on ated with performance the program and interested the performance measure standards are developed in applying. receiving the greatest em- and operate, how the au- phasis, the employment thority for writing and rate at termination. The Focus of the Manuscript implementing these con- The manuscript is orga- tracts is distributed In all cases, the ana- nized interestingly, with across various political lysts look for evidence of the clear objective of bodies, and how JTPA oper- effects that could be at- providing the reader with atives exercise substan- tributed to the JTPA per- a detailed understanding tial discretion in making formance/incentives sys- of the important steps in JTPA resource allocation tem, separate from those the JTPA process from cli- decisions. associated with JTPA’s formal service interven- ent application through Heckman and Smith then tions. service delivery to pro- look at “creaming” in gram outcomes. This pro- JTPA, studying the rela- gression serves as a tionship between well- The JTPA Selection, Assessment and framework for reviewing specified definitions of Assignment Process the rigorously conceived “creaming” and JTPA’s com- Smith’s work concen- and carefully conducted peting goals. The purpose studies of how the JTPA trates on the four sites is to determine the poten- within the sixteen sites performance standards sys- tial importance of cream- tem is likely to serve as studied in the National ing. They identify three JTPA Study, where data an unmeasured but critical major concerns: 1) ef- program intervention. were collected on JTPA-el- ficiency — in that it is igible individuals not Jeffery Smith, a faculty believed that the least participating in the pro- member at the University advantaged eligibles are gram. These data made it of Western Ontario, Cana- most likely to benefit possible to examine more da, and an associate of from JTPA and creaming may fully the “front-end” of the Center, studies four reduce their participa- the service delivery sys- of the National JTPA Study tion, 2) equity —in tem. sites to 1) document the that the program seeks

Evaluation Forum n Issue 12 n Summer 1997 73 Evaluation Issues and Activities n The Performance Measurement Movement

Smith compares the eli- adults to be conscious of they were likely eligi- gible population at the the program and their own ble. four sites with the Na- eligibility, and to have African Americans were tional Study’s experimen- applied or been partici- n more likely than whites tal groups, the latter pants in the past. These to have heard of the consisting of individuals differences may have been program; whites more who had completed the ap- related, Smith suggested, likely than Hispanic plication-through-assess- to patterns in the cost/ Americans. Among out-of- ment process. Additional benefit trade-off associ- school youths, African information was collected ated with participation, Americans were more on all program applicants to local decisions, and/or likely to be familiar at one of the sites and to local economic environ- with the program than compared with the experi- ments. whites or Hispanic Amer- mental group, to establish Smith’s major findings icans. what kinds of individuals were these: with what sorts of charac- n The low rate of program teristics were most likely n Women constituted a awareness among Hispanic to drop out in the appli- sizeable majority (65%) Americans was attributed cation-assessment process. of the eligible popula- in part to the language tion in all four sites, barrier. The main questions of and the proportion was interest in the cross-site In comparing the experi- roughly the same across comparison were: mental sample with the sites. sample of eligibles, Smith What differences exist n There were substantial finds a number of demo- in the demographic char- n differences across sites graphic differences that acteristics of appli- in the racial/ethnic may have been related to cants and enrollees? composition of the eli- patterns in the cost/bene- n What are the differences gible sample (one was fit trade-off for partici- regarding the probabili- 80% white; another 80% pants, to local decisions, ty that enrollees will Hispanic Americans). and to local economic con- actually receive train- ditions. An analysis of Over 10% of the eligible ing (i.e., with the ex- n the one site providing sample at the four sites ception of those random- systematic data for appli- had previously applied ly selected into the cants during the period of to JTPA and over half of study’s control group)? the National Study showed these applicants report- that men, local minority What are the differences ed having received JTPA n groups, and out-of-school in the types of services services in the past. youth were more likely to enrollees receive? n African Americans were complete the application more likely to have ap- and assessment process The Selection Process plied previously than than either African Ameri- In comparing the experi- whites or Hispanic Amer- can or Hispanic American mental sample with a sam- icans. Out-of-school applicants, and that ple of JTPA eligibles in youth were more likely adults were more likely to the four sites, Smith than adults to have ap- do so than out-of- school found a substantial amount plied previously. youth. of selection based on de- n Eligible African Ameri- mographic characteristics cans who had applied Assignment to JTPA Treatments at all stages in the pro- previously were more Following assessment, cess of application likely to report the re- and prior to being random- through the receipt of ceipt of JTPA services ly assigned to control vs. services. Significant de- than were previously-ap- experimental groups, JTPA mographic differences ex- plying whites or Hispan- participants in the Na- isted between the eligible ic Americans. tional JTPA Study were as- population and the group signed to one of three ma- completing this process in n Barely more than 40% of jor service interventions, terms of awareness of JTPA the eligible sample had presumably based on the and the probability of re- heard of JTPA, and of assessment information. ceiving particular kinds these just under half These interventions were: of services once in the realized they were eli- 1) classroom training in program. Eligible out-of- gible. In total, only occupational skills (CT- school youth were more 18.4% of eligible indi- OS), 2) on-the-job train- likely than eligible viduals had heard of the program and understood

74 Evaluation Forum n Issue 12 n Summer 1997 Evaluation Issues and Activities n The Performance Measurement Movement

ing (OJT), and 3) “other vention. The Incentive System services” (mainly job In the four sites, Afri- readiness preparation, job JTPA is described by can Americans had a lower search assistance, try-out Courty and Marschke as em- probability of receiving employment and/or various bodying “the spirit of the any of the three service non-training services). In New Federalism in the Re- alternatives. The differ- comparing assignments agan era” — that is, it has ential was larger for Af- across the four study a highly decentralized rican American youth as- sites, the key findings program design. The major signed to OJT. Both were these: outcomes are to be in- African American and white creased employment and n Sites differed in the adults were substantially earnings, and a reduction proportion of clients in less likely to receive in welfare dependency. A the experimental sample services than at the other distinctive feature of the assigned to each of the experimental sites. Out- program is an incentive three service interven- of-school youth in the system that rewards ser- tions. Some sites empha- four sites were more like- vice deliverers that per- sized OJT, for example. ly to receive services form well based on a set than adults, but less The four sites had a of “performance stan- n likely to receive them higher rate of assign- dards.” JTPA is, in fact, than in the other experi- ment to CT-OS and a low- the first large-scale na- mental sites. er rate to OJT and “oth- tional program in the U.S. er services” than did Routinely collected JTPA that uses financially- the remaining twelve ex- administrative data clas- based performance measures perimental sites in the sifies services into and standards, a feature National Study. classroom training, OJT, later incorporated in the job search assistance, ba- JOBS program for welfare Whites had a higher n sic education, work exper- clients and proposed for a probability of assign- ience, and non-training number of other national ment to OJT than African services. The management initiatives. Americans or Hispanic information system reports Americans. States must set aside 5% who received these servic- of their JTPA appropria- n Out-of-school youth were es. Based on these data, tion to fund this incen- more likely than adults Smith finds that 1) women tive and sanction system. to be assigned to CT-OS. were more likely than men Congress, the U.S. Depart- to receive classroom ment of Labor and the The Receipt of Services training and men more states all play a role in likely than women to re- defining performance stan- Assignments did not al- ceive OJT; 2) out-of- dards and incentive/sanc- ways predict the services school youth were more tion policies. This in- individuals actually re- likely than adults to re- volves determining the ceived. In this respect, ceive non-training servic- quantitative measures that Smith’s analysis is quite es, with adults more like- will represent key outcome enlightening. Only 58% of ly than youth to receive variables, setting numeri- the individuals completing job search assistance; 3) cal standards, and devel- the application through the study sites provided oping a model for adjust- assessment process at the less basic education, work ing standards that four study sites actually experience and non-train- accounts for influences received one of the three ing services than other considered beyond the con- JTPA service interven- experimental sites, and trol of the service pro- tions. Most of these indi- more job search assis- viders. viduals received only one tance; and 4) Hispanic of the three types of ser- Americans had very differ- In 1984 the Department vices, whereas in the oth- ent service receipt pat- of Labor established seven er twelve experimental terns than those in the performance measures — for sites 64% received one of other sites, being more youth, adults and welfare these service interven- likely in the four study recipients. In 1988 the tions. Over 73% of those sites to receive non- Department introduced new assigned to CT-OS in the training services — al- measures that included four study sites actually though this result was “post-program” outcomes, received classroom train- skewed by the site with to encourage the provision ing, whereas just over 51% the largest number of His- of more intensive train- of those assigned to OJT panic American partici- ing. In 1992 the cost mea- in the other experimental pants. sures in performance stan- sites received this inter- dards were eliminated and

Evaluation Forum n Issue 12 n Summer 1997 75 Evaluation Issues and Activities n The Performance Measurement Movement

measures to encourage pro- graphical region; there- priately and how lacking viders to serve the “hard fore, these providers can in “robustness” different to place” were added, manipulate the standards. kinds of incentive systems based on a U.S. General Also, the model may not are in avoiding “gaming” Accounting Office study hold training providers behavior. The study showed suggesting that JTPA man- harmless for serving indi- that the incentive system agers were discriminating viduals with different was likely encouraging job against certain kinds of characteristics, because training organizations to eligible clients in the it is based on provider- “waste resources in coun- selection process. level rather than individ- terproductive actions in ual-level data. In addi- order to maximize their Currently there are four tion, adjustment factors performance scores.” That core performance measures are calculated using the is, such systems may have for adults: two postpro- participant population been increasing the bud- gram employment measures rather than the eligible gets of these organiza- and two average earnings population. tions based entirely on measures; one set for all those organizations’ suc- adults and one set for These features aside, cessful gaming strategies. welfare clients. Some Courty’s and Marschke’s states have added their analysis indicates that Based on this central own measures, such as ex- federal guidelines for insight from the study, penditure measures, “sys- computing the difference the authors suggest that tem-building” measures, between actual performance the JTPA incentive system and measures of employment and desired performance is implicated in allocat- retention. (as measured by standards) ing resources dispropor- gave states and service tionately to organizations A regression model is providers substantial that may have been ineffi- used to estimate the im- flexibility in interpret- cient or ineffective vis- portance of factors out- ing the guidelines. This a-vis JTPA’s longer-term side the providers’ con- flexibility gave them lee- goals. In this sense, the trol, such as the way to manipulate the per- researchers see the poten- demographic and economic formance standards system tial for the incentive characteristics of the as well as to meet or ex- system to become “a moral providers’ geographical ceed the standards. Also, hazard,” maximizing narrow area. There are several incentive contracts speci- measures of performance at important aspects to the fying the conditions under the expense of job train- regression model. It must which financial awards ing organizations’ intend- involve measures of the were to be given varied ed objectives. independent explanatory considerably across states variables and the depen- Their conclusions are and years, suggesting that dent variable. It must that 1) JTPA organizations states used their discre- identify which variables tended to terminate em- tion in designing these have the most significance ployed enrollees at the contracts, often rewarding and why. Then problems end of their training service providers for en- must be anticipated with spells while delaying the rolling certain target respect to the assumptions termination of unemployed groups of eligibles. underlying the model or enrollees, and 2) these constraining its generali- The researchers charac- organizations terminated a ty. terize their study as of- large fraction of partici- fering “compelling evi- pants at the end of the Such problems must be dence” that JTPA accounting year. They at- given attention. For exam- organizations “game” the tribute the latter finding ple, performance standards incentive system. The to the annual periodic na- vary with the enrollment analysis provided, they ture of performance mea- and termination decisions say, the first hard data surement which was accen- of training providers on how difficult it is to tuated by performance within a particular geo- measure performance appro- thresholds. They go fur- ther, suggesting that the

76 Evaluation Forum n Issue 12 n Summer 1997 Evaluation Issues and Activities n The Performance Measurement Movement

phenomenon of maximizing gain, protection from Editorial Commentary the incentive award is criticism, and general true of any explicit in- credibility. Most of the authors of the articles, centive system incorporat- papers, monographs and books re- ing performance measures What the researchers viewed in the series on performance that are not well-aligned find is quite interesting. measurement systems have made a with the intended goals of There was a strong pattern compelling case for a results-oriented the organizations in- across fourteen of the approach for redefining government ac- volved. sixteen National Study countability — an approach that reflects sites of terminating a earlier public management concepts Using Performance Standards to large proportion of JTPA and borrows from the private sector. participants in the last Motivate Program Organizations month of the contract Some of the articles make an equal- Courty and Marschke test year, which the research- ly dramatic case for studying the impact whether JTPA bureaucrats ers maintain was consis- of using performance measures on the respond to changes in per- tent with “an award-maxi- achievement of the intended goals of formance incentives. Using mization model of job social initiatives. Clearly there is a descriptions of incentive training center behavior.” down side to the use of performance policies for each of the Based on “participant in- systems, particularly in the absence of sixteen National Study ventory control strate- evaluation research. The inappropriate sites, administrative data gies,” the researchers ex- manipulation of programs in order to on inputs, outcomes and plain this pattern in obtain incentives associated with per- enrollment/training deci- terms of the high marginal formance standards systems is an ex- sions, and a freshly-de- value in meeting perfor- ample. signed survey of JTPA man- mance standards which was Although some concern is being ex- agers and staff, the achieved by waiting until pressed, internationally, about the po- researchers study the ex- the end of the contract tential negative effects of performance tent to which the rela- year to terminate unem- systems, the U.S. has been in the fore- tively weak financial link ployed participants. front of studying such effects, as wit- between performance and The researchers find nessed by the Upjohn Institute book incentive awards was given that the nonemployed were edited by Jim Heckman. The recom- serious consideration by mendations flowing from these studies JTPA managers. The authors driving the end of the year termination pattern. should serve as a caution to policymak- give attention to the non- End-of-year terminees ers, agency administrators and pro- financial reasons bureau- gram operators. They should suggest crats may view incentive showed significantly lower outcomes on all perfor- new ways to prevent a displacement of policies useful in manag- mance measures for all policy and program goals, if we are de- ing their programs, such termined to participate in the perfor- as increased career pres- years, and enrollees ter- minating at the end of the mance measurement movement. tige, higher salaries or promotions, political contract year had longer enrollment spells than en- rollees terminating in other months. This is con- sistent, the researchers conclude, with maximizing the likelihood of an in- centive award by sorting individuals into the year in which their marginal value was highest. Although the authors suggest that further re- search is needed, they deemed definitive their tests of the validity of their hypothesis that in- centives change bureau- cratic behavior and lead to a significant change in measured outcomes. They claim that bureaucratic actors do respond to in- centives by managing their inventories of trainees in the interest of obtaining an award, and that this Evaluation Forum n Issue 12 n Summer 1997behavior in fact does lead 77 to different (reported) program outcomes. Evaluation Issues and Activities n Alternative Evaluation Approaches

Alternative Evaluation Approaches nn mates of program costs as well. A true Editorial Introduction Testing the Use of experimental design, with random as- signment to treatment and control Some experts on the evaluation of Program Applicants groups plus no nonrandom attrition or youth programs have concluded that in Studying the Net crossovers, will guarantee an unbiased evaluators, and those authorizing re- estimate of program net benefits and search, need to consider a wider range Effects of Social any foregone earnings, should they ap- of methodologies in studying the imple- ply. mentation and outcomes of these pro- Programs grams. The three articles in this section However, even the best managed ex- provide ideas about evaluating net im- periments will suffer from attrition and possibly crossover behavior. Conse- pact when random assignment is not A guest review by feasible; using a mix of different meth- quently the problem faced by the au- ods; and developing a model for under- Ernst W. Stromsdorfer thors of the Upjohn book must be ad- standing and judging complex interre- Department of Economics dressed even by those who design and lated programs. Washington State University operate experiments as well as by those who assess ongoing programs or dem- onstrations. The nonrandom selection into a program or attrition from a pro- This review article comments on a gram are problems because the selec- recent publication from the W.E. Upjohn tion or attrition behavior can be a func- Institute for Employment Research writ- tion of unmeasured and unmeasurable ten by Stephen H. Bell, Larry L. Orr, variables. Setting up an experiment John D. Blomquist and Glen G. Cain ti- may reduce the (largely unknown) ex- tled Program Applicants as a Compari- tent of the selection bias problem, but it son Group in Evaluating Training Pro- will not eliminate it, since some form of grams. This study attempts to extend attrition will always occur, human sub- nonexperimental evaluation methodolo- jects being what they are. Thus, these gy by developing statistically unbiased authors are attacking, as many others estimates of the net economic impact of have, a very difficult and significant governmentally-subsidized training pro- problem — perhaps the problem in so- grams — that is, to estimate net impact cial program evaluation. when there is no random assignment to treatment and control groups. In such In their first chapter, “Methods Used nonexperimental situations, training to Evaluate Employment and Training programs may operate by means of a Programs in the Past,” the authors set demonstration set up specifically to as- forth the history of nonexperimental sess how a particular mix of treatments methods used to attempt a correction affects a specific target group or groups, for selection bias. This chapter will be or the programs being assessed may be particularly useful to those who wish to operational programs. acquire a thorough review of the exist- ing state-of-the-art regarding these Concern over selection bias has ex- econometric procedures. isted for over two decades, but thus far a solution to the selection bias problem, Across the set of methods now used, like the Holy Grail, has eluded discov- such as the Heckman two-stage ap- ery. The present study is no exception, proach, the crucial question is whether though it is a very useful piece of work the model used to control for selection and should be mandatory reading for is statistically identified by any of the anyone who intends to understand and observed variables that are included in perform social program evaluation. the model. Put simply, the two-stage Heckman approach first estimates a The fundamental conceptual issue in probability function to predict program evaluating a training program or any participation. It then feeds the informa- other social program is the issue of pro- tion gained from this probability func- viding statistically unbiased estimates of tion into the benefit function used to outcomes or benefits of the program measure program impact. Heckman has treatment. Of course, since foregone demonstrated that in a properly speci- earnings are often involved in the cost- fied model, if there is only one variable benefit analysis of a social program — that identifies the probability function such as a training intervention — it is in the model, his procedure controls for also necessary to provide unbiased esti- the selection. Estimated net program ef-

78 Evaluation Forum n Issue 12 n SUmmer 1997 Evaluation Issues and Activities n Alternative Evaluation Approaches

fects are unbiased. Note that just add- this demonstration is a true experiment cacy of using no-shows as a comparison ing more measured variables to the that includes the use of a subjective group is in considerable doubt. This probability function in the first stage of rating index by the program’s intake group may serve as a comparison group the Heckman procedures is no guaran- workers concerning an applicant’s suit- no better than any other group of pro- tee whatsoever that identification has ability to be a homemaker-home health gram-eligible individuals. occurred. In short, the solution to the aide.The rating scale ranges from 1 (ex- To see how matters turned out in this problem is elusive since variables such cellent prospect) to 4 (poor prospect). case, the empirical results are worth dis- as motivation, IQ, or the taste for work Detailed data on measured personal cussing. The initial difference in mean are either unmeasurable or difficult to characteristics are supplemented with earnings over the four-year follow-up measure. One never knows for sure, giv- two years of preprogram earnings data period represented by the IRS earnings en the current state of statistical art, if a and four years of postprogram earnings income data — that is, experimental given model is identified in a behavioral data from Internal Revenue Service files. earnings minus no-show group earnings sense. Consequently the database contains — ranged from $1,039 (year two) to highly accurate earnings data — in the In effect, the authors argue in Chap- $1,192 (year three). Regression adjust- context of a true experiment with some ter 2, “The Case for Applicant-Based ment that included the subjective rating attrition — and an attempt to subjec- Comparison Groups,” that the statistical index reduced the postprogram gain tively assess certain unmeasurable char- identification problem is reduced if one from a low of $135 (year one) to a high acteristics that might predict program selects as a comparison group those in- of $333 (year two) But these results success. The study sample contains: dividuals who in some way are “non- were still $123 to $420 higher than the participating applicants”. The idea here n Withdrawals: persons who applied true experimental net effects, which is that such non-participating applicants but dropped out before they could be ranged from $643 (year one) to $394 are more similar to actual participants screened by program operators. This (year three). Since the value of the so- than are any other members of the pro- group contains individuals who may cial investment criterion (the internal gram’s target group, in terms of unmea- or may not have been selected into rate of return) depends on the absolute sured characteristics. (Recall that the the program. size of the cost and benefit effects over legislated target group for a given pro- time, there is the potential for consider- n Screen-outs: persons who applied gram is often very broadly defined, such able bias in the estimation of this in- but were rejected for admission to as “the long-term unemployed” or “the vestment measure if one uses the no- the program. economically disadvantaged”.) The au- shows as a comparison group. thors argue that this approach is partic- n No-shows: applicants accepted for The authors attempted to overcome ularly valuable if systematic records of participation but who failed to par- the difficulty of having to make an arbi- the observable characteristics of appli- ticipate. trary choice among these estimated cants are kept throughout each step of n Experimental participants. models and their comparison groups in the selection and screening process by Chapter 5, “Testing Alternative Methods the program managers who are selecting n True controls. for Selection Bias”. The main question program participants. To test the efficacy of their recom- addressed is “Which, if any, of the ap- They properly note, however, that mendation concerning comparison plicant-based approaches is sufficiently questions concerning unmeasurable group selection, the authors first esti- protected from selection bias to provide characteristics such as motivation, in- mated net program impacts with the ex- adequate substitutes for a randomized herent ability, or the taste for work will perimental participants and the true experiment?” The authors provide an ef- still exist. So, a priori, while it is rea- controls. Then they compared these re- fective review of the literature that fo- sonable to assume that the authors’ rec- sults against separate regression models cuses on the attempt to select an unbi- ommended procedure will reduce the that use as the comparison group the ased estimator. They then move on to problem of dealing with unmeasurable withdrawals, the screen-outs and the their preferred method, the develop- characteristics that underlie selection no-shows, respectively. One might ex- ment of a Bayesian decision framework into a program, one does not know to pect, a priori, that the withdrawals and that will allow them to distinguish what extent the problem is resolved. the screen-outs would not measure up among competing estimates of net pro- And in any case, the authors’ main as- well as valid comparison groups. This gram impact. sertion that non-participating applicants turned out to be the case. Neither were While clear, the discussion is in- are those most similar to the program’s the results encouraging for the no- volved. It will most likely escape the in- intended participants is always subject shows, even after employing in the sta- tellectual appreciation of most govern- to empirical verification. tistical models the selection rating index ment policymakers, legislators and their scored by program operators. Chapter 3 of the book, titled “Esti- staffs, based on this reviewer’s own ex- mating Program Effects in the Home- It is arguable that the no-show group perience over the past thirty years. Be- maker-Home Health Aide Demonstra- is the most comparable to the treatment ing difficult to explain and quite closely tion,” presents the initial test of the group. However, this statement depends and logically reasoned, the discussion authors’ recommendation in Chapter 2 entirely on why the no-shows do not will probably fail to meet the authors’ by using a database that has both a true participate, and on the variety of rea- own subjective test of an effective policy control group and observations about sons the no-shows do not participate, tool. In their own words, “It is not applicants who were not allowed to par- since they are not likely to be a homo- enough that an estimation technique be ticipate or who voluntarily declined par- geneous group.The reasons for selection considered free of selection bias in the ticipation at some stage in the program out of the program are key. If these rea- minds of the researchers; it must also operation process. The evaluation of sons cannot be measured, then the effi- be viewed (the emphasis is in the origi-

Evaluation Forum n Issue 12 n Summer 1997 79 Evaluation Issues and Activities n Alternative Evaluation Approaches

nn nal) as bias-free by policymakers if its Opinion and attribute studies can results are to be trusted and used.” Participatory Mixed- be conducted...Actual decision- making processes can be Even more to the point, as the au- Method Evaluation observed...but polling data and thors point out, “...even if one or more “case studies” of individual be- of the applicant-based estimators exam- havior are mistrusted in orthodox ined here passes the validation tests we Many youth experts have claimed economic methodology. The use employ, we cannot claim to have found that narrow sets of issues and singular of these kinds of data to discover an applicant-based technique that will evaluation methodologies, particularly the realities of economic life and be as reliable as an experiment in all in isolation from one another, are totally to provide an empirical basis for contexts.” The method proposed has inappropriate for studying youth devel- the auxiliary assumptions we high internal validity but unknown ex- opment, education, training and em- need will require a revolution in ternal validity. What has eternally elud- ployment issues. They insist that a methodological beliefs in econom- ed us is the discovery of an estimator broad set of issues must be addressed ics, and a revolution in the train- with a high level of external validity simultaneously, a continuum of re- ing of economists... (which allows us to generalize net im- search designs applied, and multiple pact results), or a statistical test that methods used if we are to gain an in- will identify such an estimator from depth understanding of the relationship An International Exemplar among a competing set. between 1) social program interventions The Evaluation Development and Re- Thus readers are left with a sense of and 2) program outcomes and longer- view Unit of The Tavistock Institute in frustration in assessing the authors’ ef- term effects in social policy initiatives London, directed by Elliot Stern, who forts. Nevertheless, despite this continu- targeted to adolescents and young organized the United Kingdom’s evalua- ing problem, the monograph is a very adults. This point of view stems from tion association, has been applying par- thoughtful and well executed analysis. the belief that the interaction between ticipatory mixed-method evaluation ap- It should be required reading for every- socialization experiences and institu- proaches for over twelve years. What one performing program analyses, for tional interventions are exceedingly the Tavistock researchers have done is those attempting to interpret the results complex — young people are in a period to weld together, with strong bonding at of such analyses, and for those who de- of self-image and behavioral evolution, the front end of the evaluation process, sign new social initiatives. vulnerable to an array of influences and the following ingredients: 1) the princi- choices. Consequently these experts ples and practices imbedded in plan- propose that the evaluation of youth ning, systems analysis, organizational programs must address a wider array of theory and human resource develop- developmental and other issues, ment, and 2) their research training, through the use of a more “participa- knowledge and expertise. What further tory,” “mixed method” type of research. distinguishes their approach from tradi- tional practice is the synthesizing of the Even Nobel laureates in Economics norms, values and ethical contexts that have given attention to less mainstream support these other disciplines. This approaches. For example, Herbert Si- borrowing, marrying and integrating of mon, in The State of Economic Science: different perspectives has led them to Views of Six Nobel Laureates, published embrace a collaborative, supportive, in 1989 by the W.E. Upjohn Institute for mixed-method approach to evaluation. Employment Research said: A new wave of American evaluation A very sophisticated body of researchers also have emphasized the econometric theory and method- importance of the relevance of an evalu- ology has been created since ation to its policy context; the appropri- World War II to deal with the ateness of an evaluation plan to the problem of carrying out economic goals and developmental phase of the analyses with data that are too ag- program being evaluated, and to its en- gregated, measure the wrong con- vironment; and the competence and ob- cepts, are too infrequently collect- jectivity of the evaluator’s methodology. ed, and are exceedingly noisy. The An appreciation of the political, socio- experience of the past 50 years cultural, economic and organizational has shown that these sophisticat- environment of an evaluation effort is ed econometric tools do not com- viewed as a critical asset. An ability to pensate for the poor quality of the invest key players from the beginning to data to which they are applied. the end of the evaluation planning, im- plementation and utilization process is ...Other kinds of data can of understood to be essential in encourag- course be obtained: data that tell ing and sustaining support for an evalu- us something about the expecta- ation activity. It is considered signifi- tions, preferences, and choices of cant in assuring that evaluation results economic actors at micro levels. and recommendations are communicat- ed honestly, communicated at the right

80 Evaluation Forum n Issue 12 n SUmmer 1997 Evaluation Issues and Activities n Alternative Evaluation Approaches

decision points in the policymaking pro- vest program practitioners and their up of research funding may eclipse cess, and communicated to those in a stakeholders in the evaluation pro- this problem altogether. position to initiate, rationalize and cess, increase the often troubled 5. The Tavistock approach is demand- maintain change.This is a stakeholder, working relationship between re- ing of multiple skills and talents, and user-oriented evaluation process that searchers, program staff and others quality control and information inte- demands an eclectic tool kit. invested in the program being evalu- gration are large challenges not all ated, and increase substantially the A side benefit of this approach, not evaluation teams could manage. The use made of evaluation information insignificant, is what Americans now potential cost of this interdisciplinary for continuous program improvement like to call “capacity-building.” Building expertise is also a consideration, par- and policy change. greater evaluation capability recognizes ticularly the excellent introduction of and appreciates the differences in the The concern, however, is that it is a educational training seminars, some- assumptions, values, knowledge base, sizable challenge to maintain suc- thing all researchers should try to in- training and experience of the diverse cessful coordination between 1) self- corporate in evaluation efforts. constituencies involved in most evalua- evaluation activities and results, and 6. The Tavistock approach involves tai- tion efforts, and seeks to reduce these 2) evaluation activities conducted by loring a unique mix of organizational differences by encouraging the partici- external and program-independent and research strategies to the priori- pation of diverse actors while providing researchers and their results. This re- ties given attention by policymakers, them with an education about the re- quires an external team with the and to the nature and development search process. funding, time and diverse expertise stage of a social initiative — some- to assure that joint planning at the In the wake of an unprecedented de- thing researchers frequently fail to front, collaborative implementation centralization of social programs across do in requesting and launching eval- in the middle, and an integration of all postindustrial nations in the context uation projects. results and recommendations at the of new economic restraints, increasing end are held together. Where the 7. There are valid concerns about mak- local and regional capability and trust needed resources for useful self-eval- ing comparisons across similar pro- are now mandatory. The pay-off is new uations appear to be unavailable, grams at the local, regional, national possibilities at those levels for exploring where the management of this com- and transnational levels, as part of subnational questions of direct interest plex an evaluation effort is likely to the mix of methods. This is a very and concern to people responsible for be weak, or where the independent tricky business. One must reduce operating and modifying programs. evaluators commissioned for the bias in such comparisons by identify- study are likely to take inappropriate ing the criteria to be used in judging Strengths and Weaknesses control of the entire effort, there are the true comparability of the pro- Stern commented about the potential problems. grams to be compared. Important is a prior description of conditions such weaknesses, as well as the clear 3. The importance of developing a new as the following: strengths of the more pragmatic mixed- “evaluation culture” cannot be over- method approach, in his session at an emphasized. This is what the Tavis- – the policy context international evaluation research confer- tock approach is all about. It requires – the program environment ence in Vancouver, British Columbia in a redefinition of the traditional roles November 1995. The following com- of policymaker, program operator, – the intended program goals, ments by the editor of Evaluation Fo- stakeholder and researcher in the means, interventions rum, Ann Bonar Blalock, echo some of collaborative, information sharing di- – the intended population to receive Stern’s anxieties and suggest others: rection — a long overdue revision of the interventions 1. This approach is more appropriate evaluation researchers’ usual style and useful in evaluating innovations and inclination. – the desired outcomes for this pop- ulation in their early stages, because one 4. The Tavistock researchers point out wants maximum flexibility in this that the relationship between 1) pro- – the actual characteristics of pro- case for identifying key variables that gram safety in a hostile world, and gram implementation should be measured and studied in 2) the willingness of program per- – the actual characteristics of the later evaluations. But this approach sonnel to resist the demand to “look target group should not be precluded in studying good at all costs” is a significant one ongoing programs that have a more in this era of economic scarcity and – the actual interventions received established track record, because it government downsizing. But one – the characteristics of program expresses a commitment to looking wonders if the collaborative mixed- management and service delivery at both program implementation and method approach can survive this and the nature of the information effects. Large-scale, periodic evalua- onslaught, since program staffs tend collection, storage, and analysis tion efforts with ongoing programs to feel now like the workers caught process could clearly benefit from a similar in a web of potential mergers and emphasis. mass layoffs in the corporate sector. – the research methods used and by 2. It appears to be true that internal If evaluators can be seen as safety what kinds of researchers mechanisms rather than as danger self-evaluation efforts (part of the The problem is that one cannot zones, there will be continuing sup- participatory approach) that are tied take comparability for granted. The port for a better approach — but this carefully to less subjective indepen- title of the project or program may remains to be seen. And the drying dent evaluation activities can best in- be the same from site to site, but

Evaluation Forum n Issue 12 n Summer 1997 81 Evaluation Issues and Activities n Alternative Evaluation Approaches

there are always significant differ- cal to the evaluation. The treatments authors propose that participatory eval- ences in the way an abstract pro- are often poorly defined. And it is uation is an extension of the traditional gram is given life in a particular en- difficult to interpret the results of ex- stakeholder-based model. Key charac- vironment. These differences render periments because net impact stud- teristics are: sites less than comparable. Conse- ies are frequently authorized in the quently evaluators must try to identi- absence of an effort to study the n Control of the evaluation project is fy and compensate for these differ- characteristics of program implemen- jointly shared by researchers and ences in comparing the tation that may explain program ef- practitioners. implementation and effects of these fects. Even highly accurate informa- n There is a limit to the number of different applications of a central tion about outcomes can be stakeholders involved. concept, or of programs that appear misleading in the policy process, if to flow from the same premises and one has no idea why or how they n There is an unusual depth of partici- serve the same general purposes. happened. pation among non-researchers. 8. Not apparently included in the mix 9. In this complex an effort, researchers In the traditional model, stakehold- of methodologies within the Tavis- must keep track of who, with what ers are consulted early to define the fo- tock approach are research designs potential set of biases, is collecting cus of an evaluation and later to help which estimate the net effect of a information about what and by what interpret the data, but they are only program’s unique set of interventions means, so that the evaluation team rarely involved during the research pro- — these experimental and quasi- holding this all together can distin- cess and write-up phase. In the partici- experimental evaluation designs are guish between different levels of sub- patory model, joint control of the evalu- generally considered “state of the jectivity in the information to be ation means that the evaluation agenda art” impact methodologies by a large brought together at the end. This is is mutually determined and conducted. cohort of American evaluation re- critical in interpreting information Stakeholder participation, however, is searchers. The ability to estimate the from multiple sources and therefore limited to those with program responsi- net return on the public’s investment in making policy and program rec- bility or an investment in the program. in a program has appealed to policy- ommendations based on this infor- Members of the program community makers in this market economy for a mation. are involved in defining the evaluation, good reason — it allows them to developing instruments, collecting data, place evaluation results within a fa- processing and analyzing data, and re- miliar investment/profit framework. porting and disseminating results. This Editor’s Note involvement of practitioners and stake- But these designs are not perfect, holders is expected to increase the util- therefore their absence in the mixed Elliot Stern’s paper at the interna- ity of an evaluation and the capacity of method approach may not represent tional conference, written with Elizabeth those participating to organize and car- a research deficit. Such methodolo- Sommerlad, was titled “The Devolved ry out useful evaluations. gies limit the number of variables Evaluation of Dispersed Programmes.” that can be studied, and the number Stern is the editor of a new internation- The senior author of the article, J.B. of measures that can be used to de- al evaluation research journal pub- Cousins of the University of Ottawa, fine those variables. They alter the lished by Sage titled Evaluation. Canada, has written extensively on this approach to evaluation. He also has intended implementation mode of a An interesting article in the fall 1995 program that may distinguish it from written two papers for international re- newsletter of Harvard University’s Fami- search conferences which may be of in- other alternatives and help explain ly Research Project, The Evaluation its effects. It is difficult to convince terest: “Collaborative Evaluation: A Sur- Exchange, discusses an approach sim- vey of Practice in North America,” and program staff that maintaining ran- ilar to Tavistock’s in “Participatory Eval- dom assignment procedures is criti- “Consequences of Researcher Involve- uation: Enhancing Evaluation Use and ment in Participatory Evaluation. “ Organizational Learning Capacity.” The

82 Evaluation Forum n Issue 12 n SUmmer 1997 Evaluation Issues and Activities n Alternative Evaluation Approaches nn vide insights that could be incorporated Partnership Collaboration and The European by the Center within a general evalua- the Goals of Intermediate tion guide for use by the Monitoring Commission’s Effort Committees the EC had established to Evaluation to More Effectively be formally responsible for planning The EC guide places strong emphasis and oversight regarding the projects on a heightened sensitivity to the needs, Structure Evaluation supported by the Structural Funds. interests and resources of multiple stakeholders, and the creation of a col- Activities These Monitoring Committees are laborative partnership-oriented environ- unusually large, composed of represen- ment — a new evaluation culture. It tatives of the EC, government officials differentiates between “mature” part- The directorates of the European in the host member countries, and nerships characterized by shared objec- Commission (EC), the central govern- Structural Fund project managers and tives and combined budgets; “single ac- ment of the European Union, carry out program operators in the areas served tor-dominated” partnerships that are the policies developed by the executive by the Committees. The staffs support- more hierarchical; and “loose ad hoc” bodies of the Union, the Council of Min- ing these committees have not, in most partnerships that survive on weak con- isters and the European Parliament. A cases, included personnel with research sensuses. It asks that Monitoring Com- significant proportion of the Commis- training or experience. Consequently the mittees appreciate these differences in sion’s funds are the Structural Funds Committees’ mandate to evaluate has developing stronger partnership net- for large-scale economic development, been unevenly honored, with each works. Expanding the network further, employment and training, and other hu- Committee tending to impose its own the guide suggests that Monitoring man resource projects and programs definition of evaluation. Committees include representatives of across member states of the Union. Al- The intent of the new guide was to the recipients of Structural Funds though the Commission has required increase the coordination of evaluation projects. Given this broad focus on col- that the use of these funds be moni- approaches, policies and activities with- laboration, the guide describes a dual tored and periodically evaluated, the in- in the Monitoring Committees around a purpose for evaluation activities: formation made available to the EC standard set of principles and recom- project/program improvement, and the from past assessments has not been mended practices. This standardization establishment of accountability with viewed as fully adequate for decision- of a core understanding of the value funders, partners and program recipi- making. and nature of Intermediate Evaluation ents. In an environment of increasing con- was expected to assure more efficient Because of variability in the compo- cern about establishing the European and effective studies of projects, and a sition of Monitoring Committees, the Commission’s accountability in a de- greater use of findings in improving the guide outlines both 1) basic Intermedi- volved and decentralized social provi- results of initiatives that flowed from EC ate Evaluation requirements, and 2) op- sion system, the Commission recently policies. tional recommendations. sponsored a special project to enhance Following the seminars, the begin- The basic requirements suggested are the evaluation of projects under the ning guide, Organizing Intermediate these: Structural Funds. This effort emphasizes Evaluation in the Context of Partner- the concept of Intermediate Evaluation ships, was published in late 1995. The n A clearly defined five-year evaluation — the ongoing and systematic evalua- word “partnership” has special mean- plan. tion of these projects and programs, as ing, since the European Union (EU) in- n The development of relevant targets compared with previous periodic, often volves a complex set of vertical and hor- to be accomplished over the five one-time-only evaluation efforts. The izontal partnerships pursuing common years. Center for European Evaluation Exper- goals, which are reflected in the struc- tise in Lyon, France won the contract to ture of the Monitoring Committees. In n The specification of the outputs that organize this project. this sense, the EU is a great experiment have already been achieved. in testing the viability of building coop- n The evaluation of the impact of the The Development of a New Set erative relationships among powerful “evaluable” elements of projects. of Guidelines for Evaluation sovereign partners. The Monitoring Committees are a potentially exciting n The development of recommenda- The Center sponsored several semi- test of collaborative evaluation planning tions for corrective action or enhance- nars in Brussels in 1995 on evaluation and utilization. ment. planning, design, implementation and utilization. The seminars brought to- In this context, the evaluation guide It is acknowledged in the guide that gether a select group of experts on is expected to raise partners’ awareness the level of rigor of impact evaluations evaluation research and project admin- of effective strategies for organizing and will inevitably be influenced by the in- istration to advise the EC — that is, managing competent evaluations with terests, timelines and resources of par- evaluation research experts from the high utility. The guide does not, howev- ticular Monitoring Committees. Howev- U.S. and member countries of the EU, er, suggest that an incentive/sanction er, there is no specific discussion of administrators of relevant EC director- system be developed to secure compli- different kinds of outcome evaluations, ates, and researchers on contract to the ance with the guide’s positions and rec- such as simpler pre-post studies, or EC. The purpose of the presentations ommendations. more complex quasi-experimental and and structured discussions was to pro- experimental net impact studies. Nei- ther are there cautions about undertak- ing comparative studies across projects or sites where it is difficult to obtain

Evaluation Forum n Issue 12 n Summer 1997 83 Evaluation Issues and Activities n Alternative Evaluation Approaches

comparable baseline and outcome data. about the major aspects of projects and Committees in honoring their oversight However, the guide seems to encourage programs in a collaborative, inclusive responsibilities. Entities such as these, Monitoring Committees to move toward atmosphere that supports utilization. linked directly to evaluation, can pro- rigorous net impact studies. vide advice at the beginning of evalua- tion planning, review the progress of The guide distinguishes among three Developing a Framework for evaluation activities, suggest solutions kinds of evaluations: ex ante, intermedi- Evaluation to research and organizational problems ate and ex post studies. In the European Monitoring Committees are encour- that arise, and contribute insights about vernacular, ex ante studies typically aged to develop a general, pragmatic, ways to present findings for use by dif- provide baseline information at the be- written “evaluation mandate” that clari- ferent audiences. Defining lines of au- ginning of a project or program and in- fies a particular evaluation’s orientation thority and communication, and the formation about their early implementa- and priorities and the responsibilities of roles to be played by such bodies, are tion. Ex post studies look at short and/ key actors. Apart from organizing evalu- considered a critical task. or longer-term outcomes at the end of a ations more effectively, preparation of a project’s or program’s official life span. mandate is expected to encourage dis- Intermediate evaluation represents on- cussions across the vertical and hori- Identifying Program Assumptions going evaluation of a project or program zontal partnerships within Committees. and Objectives in Developing the over its administrative or legislative life, Such discussions are intended to moti- Research Questions and is expected to contribute informa- vate members to commit themselves to One of the most difficult early evalu- tion on the basis of which competent ex the objective of an evaluation, to a gen- ation tasks is addressed in the guide — post evaluations can be conducted. Oth- eral set of research questions, to who is identifying the hierarchy of objectives er distinguishing features of ongoing responsible for organizing and carrying and intended cause-effect relationships evaluation in the guide are the length of out an evaluation, and to ways in which that define a particular project or pro- time allotted for the evaluation process, the results can be disseminated and gram. The explicit and implicit hypothe- the close association of evaluation activ- used. Evaluation design seminars or re- ses that are revealed in the legislation or ities with the monitoring process, and treats are recommended as ways to directives initiating social programs the methodological difficulties in study- structure this planning effort. tend naturally to be modified as these ing longer-term project or program ef- Key issues to be decided are opera- programs are operationalized. Conse- fects prior to the end of the life span of quently the development of evaluation the initiative. tional responsibility, the adequacy of the evaluation budget, the timing of questions must give attention to both evaluation activities, the composition the original program design and the op- Steps in Carrying out a New and roles of the evaluation team to be erational program. The guide suggests Evaluation Mandate selected, and where and when the re- that the results of such negotiations The guide is organized as an ideal- searchers are to be brought into the over goals, objectives and desired out- type progression of steps. Monitoring evaluation process. comes should be incorporated within Committees are expected to mirror Requests for Proposals (RFPs). Having reached agreement on a man- these stages at their own pace and ac- date, Monitoring Committees are en- cording to their own assessment of The Selection of Evaluation couraged to work toward a consensus what steps can occur in logical order on a core set of clearly conceptualized, Approaches and Methodologies versus simultaneously. The overarching realistic, “evaluable” questions, while The guide recommends that the bod- principle is that evaluation activities appreciating the diverse interests and ies under, or associated with Monitoring should be an integral part of Commit- agendas of members. These questions Committees that are dedicated specifi- tees’ ongoing responsibilities rather can involve studies of intersecting poli- cally to evaluation, assume responsibili- than a sometimes add-on. The steps are cies and programs, or studies that seek ty for developing RFPs for evaluations these: to isolate the influence of a particular and selecting evaluation teams with the n Securing the evaluation mandate and set of interventions. The qualification is approval of the Committees. The RFP organizational support for evalua- that the choice should be made explicit development process is to resist promis- tion. at the front-end of evaluation planning. ing too much information about too many issues, given evaluation budgets, n Setting priorities among evaluation timelines and expertise. In this develop- issues and developing RFPs. Developing an Organizational ment process, Committees are encour- Environment for Evaluation n Developing specific evaluation plans. aged to consider 1) what kinds of evalu- Some Monitoring Committees cur- ators are needed for a particular n Managing evaluation activities and rently leave evaluation planning to their evaluation, given the research questions maintaining their quality. staffs or to a group of program manag- negotiated and incorporated in the ers. The guide strongly recommends n Disseminating evaluation results RFPs, and 2) the need to contract with with the goal of utilization by key ac- that a new partnership needs to be researchers whose training and experi- tors. formed, involving members of the Mon- ence combines research expertise with a itoring Committee, their staff, and an substantive knowledge of the type of These steps highlight features of on- independent evaluation team. Creating project to be evaluated. going evaluation efforts that are intend- an evaluation subcommittee, evaluation ed to produce unbiased information task force, or research advisory group is recommended, to assist Monitoring

84 Evaluation Forum n Issue 12 n SUmmer 1997 Evaluation Issues and Activities n Alternative Evaluation Approaches

The Use of Indicators ship between evaluators, program staff grams within them. Again, evaluators and a program’s information system and Committees must work together on The EC has emphasized the use of personnel. This is expected to increase these issues. “indicators” in making assessments of the accessibility and use of monitoring projects, and the guide includes impor- information in evaluations, depending tant suggestions about the use of such Managing and Reporting on the on data coverage and reliability. But the Evaluation Process indicators. It suggests, for example, that guide clearly reserves the design of indicators must be key measures of ma- analyses and the conducting of these The guide terms the management of jor relationships expressed in a pro- analyses for the evaluation team. evaluations “a process of dialogue and gram’s design — therefore it is impor- adjustment.” This collaborative mes- tant to develop measures of variables sage is clear in these suggestions con- describing program implementation as Developing Evaluation Plans cerning the implementation of an evalu- well as those representing priority out- The guide clarifies that Monitoring ation: comes. It is recognized that measures of Committees should resist turning over some of these key variables may not be the responsibility for developing the n Systems need to be developed that available to evaluators through adminis- plan for evaluating a particular set of provide ongoing information about trative data systems, and therefore must programs or projects exclusively to the the progress of an evaluation to key be defined and collected by the evalua- evaluators selected. Rather, close com- actors. tion team. Some of the cautions in the munication and collaboration are rec- n Assurances are needed that the eval- guide regarding the development and ommended between the Committee and uation is being implemented by the use of indicators are these: evaluators as the evaluation team trans- senior evaluators selected, rather lates the content of an RFP into a spe- than personnel under them. n A careful review of program objec- cific evaluation plan. A clearly defined tives can assist Monitoring Commit- plan will consider the interests of the n Communication channels between tees in deciding what indicators are various partners as well as the goals of the evaluators and the sponsors of critical to use in a particular evalua- the program to be studied, which then evaluation must remain open and tion. engages stakeholders in utilizing evalua- utilized throughout the evaluation process and the writing of progres- n An effort should be made to develop tion results. measures of both shorter-term and sive drafts of evaluation reports. Again, in developing evaluation longer-term outcomes, and both nar- plans, the guide recommends that Moni- Information objectivity is a priority rower and broader effects. toring Committees and their evaluation in the guide. It identifies the desirable characteristics of evaluations and their n Using both qualitative and quantita- bodies consider both implementation tive indicators is important, particu- and outcome issues. This is consistent reports as the following: larly in studying the quality of out- with the guide’s emphasis on beginning n Reliability: are the data used in the comes and effects. with what hypotheses and assumptions evaluation reliable and is the analysis are imbedded in a program’s design. Absent from the discussion of indicators of these data objective? The cause-effect relationships these hy- is any reference to the use of perfor- potheses propose are often neglected in n Methodological appropriateness: mance standards, other than the inter- singular evaluations of outcomes or im- why was this methodology selected est expressed in general (performance) pact. Monitoring Committees may ob- and does it remove potential sources targets. tain accurate information about such ef- of bias to the extent possible? fects, but with little knowledge of why n Report clarity and coverage: does The Independence of Evaluators or how they occurred. Consequently the the evaluation report explain the The guide gives appropriate attention results obtained may be misinterpreted, methodology as well as the results, to the issue of research independence, and critical information for improving and in a concise but full and accu- cautioning Monitoring Committees to programs left unidentified. rate manner? select evaluators who are not linked to Because each major EC Structural n Separation of results from action the management of the program to be Funds project typically involves a set of recommendations: are recommenda- studied, have not been involved in its programs within it, Committees are tions clearly distinguished from the design or implementation, and were not faced with decisions about whether to reporting of evaluation results? part of applying recommendations from mount ongoing evaluations of an entire previous evaluations. This is an impor- project, to give special attention to cer- Dissemination and Utilization tant caveat since a substantial portion tain programs, or to compare programs of EC evaluations have been internal within a project or across program sites. The guide is unique in its recommen- evaluations conducted by EC or member The guide suggests that Monitoring dations for using evaluation information country civil servants in EC directorate Committees use a number of criteria in to improve programs while establishing evaluation units or program divisions. resolving this dilemma, such as the credibility with multiple constituencies. While supporting the idea of external project’s stage of development and the These are the suggestions: evaluators, the guide recognizes the sig- potential utility to the Committees and n Different kinds of dissemination nificant benefits in developing and the EC of obtaining information on en- maintaining a positive working relation- strategies should be developed, re- tire projects as opposed to specific pro- sponsive to the interests and con-

Evaluation Forum n Issue 12 n Summer 1997 85 Evaluation Issues and Activities n Alternative Evaluation Approaches

cerns of the different groups of part- Comments on the EC Guide evaluations in making policy decisions ners, without compromising the ac- and improving programs funded jointly curate presentation of evaluation re- The handbook produced by the Cen- by member nations of the EU. sults and the methods used to obtain ter for European Evaluation Expertise However, the guide’s use is not limit- them. For example, Monitoring Com- represents a major new direction in as- ed to Europe. It has practical application mittees may want to format the eval- sessing the EC’s Structural Funds at the state level in the United States. Its uation information differently for EC projects. Because these funds are con- insights are relevant to the evaluation of directorates and member states, tributed and spent in the context of a the new statewide human resources sys- members of the Committees, pro- powerful supra-national environment — tems which are being developed in de- gram administrators and managers, one characterized by the necessity to volved and decentralized environments stakeholders and the public -- and accommodate diverse national, regional involving multiple partnerships. Many may wish to phase in this series of and local interests and organizational of the same evaluation issues and di- reports. agendas — this beginning effort to sys- tematize and make evaluation thinking, lemmas addressed in the guide are be- n The distribution of evaluation reports planning, implementation and use more ing dealt with across American states. should be accompanied in some cas- effective is a dramatic development. The publication’s major contribution es by opportunities for providing in this respect is its emphasis on both feedback from potential users, such The Center, directed by Eric Monnier, scientific competence and partnership as small conferences, seminar or dis- organizer of the European Evaluation collaboration throughout the evaluation cussion groups. Association, has produced a clear, non- technically written, artfully summarized process. It projects a “best practices” n Dissemination can also be accom- set of guidelines covering key issues in model of a participatory, collaborative, plished through video presentations, social program evaluation. It is appro- mixed-method approach to evaluation press conferences and other non- priate particularly for the multiple part- research that clearly has implications written means. nerships involved in sponsoring, orga- for the American experience. nizing, contributing to and using

86 Evaluation Forum n Issue 12 n SUmmer 1997 Resources Resources

nn the project for review and discussion at Education and Labor to conduct a na- U.S. Youth Initiatives a national symposium funded with the tional evaluation of the systems devel- assistance of The Rockefeller Founda- oped under the legislation. In 1995, and Program tion. In 1987, a major report was au- Mathematica Policy Research, Inc. Evaluations: thored by J. Smollar and T. Olms titled (MPR) was selected as the principal re- Young Unwed Fathers: Research Review, search firm to design and conduct this Selected Examples Policy Dilemmas and Options.The report evaluation. drew these conclusions: The evaluation is guided by four key n Unwed fathers need to be held re- objectives: In total, a vast amount of attention sponsible for their children and re- n The assessment of states’ progress in has been directed toward the resolution quired to fulfill the minimum obliga- creating systems that change the way of youth education, training and em- tions of fatherhood — legally youth are educated and prepared for ployment problems in the form of poli- establishing paternity and paying fi- careers. cy analyses and evaluation research. nancial support. But public policy has frequently ig- n The identification of promising prac- n Unwed fathers should be encouraged nored the information produced. In the tices and barriers to developing new to develop a personal relationship late eighties and during the nineties school-to-work systems. American analysts often turned to Euro- with their children. pean models, which have invigorated n The determination of the nature and n These fathers’ needs and rights must extent of participation in partner- American strategies for addressing be balanced against their responsibil- ships and programs developed by youth problems. ities. employers, schools, students, public However, in this article we focus on n These fathers need assistance and programs, educational institutions selected examples of American analysis encouragement in fulfilling parental and/or others. and research that have attempted to uti- responsibilities in terms of acquiring n The measurement of the outcomes of lize the information accumulated from occupational skills and employment. multiple sources in evaluating youth students in high school, postsecond- programs. Readers interested in more n Acquiring skills and employment ary education and training, and em- information on youth initiatives in edu- benefits fathers, children and society. ployment. cation, training and employment may n Different sectors of society, and dif- The evaluation plan to be imple- access additional examples later in the ferent levels of government, must mented between 1996 and 1999 re- Resources Section. work together in encouraging re- sponds to these objectives through three sponsible parental behavior. major components: Congress and the Federal n A mail survey of local school-to-work Agencies School-to-Work partnerships, to collect data on: The Departments of Education, La- The School-To-Work Opportunities Act – Composition of partnerships. bor, and Health and Human Services of 1994 responded to years of concern (HHS) have sponsored an array of about youth development, education – School-based and workplace- youth initiatives and have supported and employment issues in the U.S. It based activities involved. sophisticated evaluations of some of benefitted from expert opinion and re- these demonstrations. We highlight two – Linkages to postsecondary op- search on youth issues in the U.S., and tions. initiatives: a pilot project sponsored by numerous site visits to European Union Health and Human Services, and the countries whose youth programs have – Approaches for assessing stu- school-to-work legislation involving Ed- provided important lessons for Ameri- dents. ucation and Labor. can youth programming. The Act pro- – Levels of student participation. vides federal funding to develop state Health and Human Services and local systems for assisting youth in – Aggregate measures of transitions In 1986, the Family Support Admin- making a successful transition from sec- out of high school and into post- istration within HHS sponsored The ondary education to further education secondary options. and skills training, and then into pro- Young Unwed Fathers Project. The n Case studies: a sample of seven to project responded to the need to review ductive careers. Federal “seed money” nine states and forty-two local part- existing information about the unwed has been received by eight states in nerships, to collect data on: father population, in the context of a 1994 and nineteen in 1995. States re- clear relationship between the skill lev- ceiving seed funds are expected to – Approaches in developing school- el of young men and fatherhood. Ten award subgrants to local school-to-work based, workplace-based and con- policy analyses were commissioned by partnerships. necting activities. The Act requires the Departments of – The availability and utilization of

Evaluation Forum n Issue 12 n Summer 1997 87 Resources

the opportunities developed for were partnership development, involv- ment of Labor published the results of students. ing employers, and linking curricula the roundtable in Evaluating the Net Im- across diverse areas. pact of School-to-Work: Proceedings of a n A survey of students using question- Roundtable. naires and a telephone follow-up: The most important barriers to devel- thirty-two randomly selected partner- oping school-to-work systems were a ships in the same states in which lack of understanding, knowledge and Public/Private Ventures, Inc.(P/PV) case studies will be conducted, to expertise regarding the development of Several years prior to the project collect data on three cohorts of 12th these systems; bureaucratic inertia, re- sponsored by HHS, Public/Private Ven- grade students at three measurement sistance and conflict; and difficulty in tures, Inc. designed the Young Unwed points between 1996 and 2000 re- creating and sustaining cooperation and Fathers Pilot Project, funded by the garding: collaboration. Similar obstacles were Charles Stewart Mott Foundation and encountered at substate levels.The – Changes in students’ access to other contributors. It was operated in twenty-seven states receiving the imple- and participation in school-based six sites across the country. Program- mentation grants viewed the national and workplace-based activities ming in the sites expressed five key ele- office as helpful and supportive, but ad- under the new systems developed. ments: education; occupational training vocated a more coordinated, consistent through JTPA; “fatherhood develop- – Changes in high school, postsec- and persistent campaign for school-to- ment” activities; and counseling and ondary and employment out- work systems by key national leaders, ongoing support, including services comes achieved. and urged more involvement by em- eighteen months beyond job placement. ployers. The Act was clear about requiring The evaluation of the pilot project in- both a process (implementation) study In February, 1996, the Departments dicated that the Fatherhood Develop- and an effectiveness study. In May, of Education and Labor sponsored a ment curriculum developed by P/PV 1996, MPR produced their initial report roundtable titled Net Impact Evaluation was a successful element of the pilot, on the School-to-Work Opportunities of School-to-Work: Exploring Alterna- but it also revealed these challenges: Act systems, titled School-To-Work Im- tives. Six papers were commissioned for plementation Progress: The State Per- the roundtable which addressed issues n JTPA eligibility requirements, based spective in Early 1996, written by Alan surrounding the development of a net on households and income, some- Hershey, MPR’s project director, and impact design. Broad areas of agree- times acted as a barrier to the enroll- Linda Rosenberg. The report covered a ment emerged from the day-long ses- ment of young fathers. number of process evaluation issues sion with school-to-work experts and n The lack of stipends frequently emerging from discussions with direc- researchers: forced participants to find jobs pre- tors of school-to-work systems in the maturely, simply to meet short-term n A net impact evaluation should not twenty-seven states receiving imple- attempt to measure the impact of the economic needs. mentation grants. Among these issues entire system, but rather the impact were: n The project’s requirement that partic- of components of the system and the ipants establish legal paternity tend- n The nature of state and local partner- interaction among these components. ed to retard recruitment. ships. n In studying the school-to-work sys- P/PV’s policy analysis, programming n The nature and extent of partnership tem in this way, there was support and evaluation activities go well beyond resources. for using a typology of state/local fatherhood issues, however. They em- systems based on a number of char- brace a range of youth development, ed- n How “development” and “implemen- acteristics. tation” grants are being utilized. ucation and employment problems. Its organizational priority has been to n The research design should include a n The funding process for local part- study of implementation (already in address “two fundamental weaknesses nerships. progress), a quasi-experimental and in youth policy: 1) a lack of program- matic attention to youth’s basic needs n The priorities set in developing sys- observational study of outcomes, and tems. a small number of randomized trails for sustained adult attention, for safety, to test either planned variations in for recreation and learning activities, n Implementation “best practices.” the system or basic assumptions un- and for work that holds future promise for labor market success, and 2) a lack n Problems and successes regarding derlying system development. the involvement of employers. of evaluation approaches that capture a n The study of system outcomes youth’s developmental progress.” n The role of the national School-To- should include the impact on em- Work Office. ployers, through a special survey or The firm has developed a number of some other research strategy. curriculum modules as part of its pro- The researchers found that the major gramming goals, such as the 25-session areas in which states felt they had made School-to-work systems were viewed Fatherhood Development module, the substantial progress were creating col- as involving more than the conventional 60-session WORKmatters module, Work laborative relationships, developing ca- array of outcomes, to encompass and Society, Crosswalks, and Life Skills reer guidance activities, building local whether youth begin to formulate con- and Opportunities. P/PV has also con- partnerships, creating a common vision crete career plans, and whether employ- ducted an experimental evaluation of for the school-to-work transition, and ers restructure their career ladders and Big Brothers/Big Sisters programs that marketing that vision. At the substate offer higher wage jobs to high school suggested that adult/youth mentoring level, the key areas of improvement graduates. In June 1997, the Depart- relationships can reduce dramatically

88 Evaluation Forum n Issue 12 n Summer 1997 Resources

initial drug use by teenagers in single- society in that new apprenticeship strat- pletion for teens enrolled in school parent families, and improve school per- egies were needed. While including when they were determined eligible for formance and peer and family relation- components common to European mod- the program, but little gain for teens ships. Beginning evaluations of the els, such as workplace-based learning who had dropped out prior to eligibility Bridges to Work and WorkPlus demon- and credentialing, alternatives such as determination. Overall, only a segment strations suggested it was possible to in- the academies and Tech Prep seemed of the program population experienced volve employers in enriching the work more appropriate. positive outcomes. experience and career potential of poor MDRC found that the workplace- youth. And the firm’s work with neigh- based training component was the most The Carnegie Corporation and borhood groups in initiatives such as significant problem area. Involving the Southern Education Plain Talk and Community Change for American employers in a systematic Youth Development clarified that ade- Foundation way, to support apprenticeship models quate resources and strong adult com- In 1995, the Corporation’s Council over time, has proven difficult. School- mitment had a significant influence in on Adolescent Development produced a based enterprises and cognitive appren- poverty areas. major report titled Great Transitions: ticeship demonstrated potential as op- Preparing Adolescents for a New Centu- In 1992, P/PV conducted a study of tions, but the linkage between these ry. Its recommendations fell heavily on a set of projects sponsored by the firm options and the workplace have re- the side of stronger roles for the em- and others which were focused specifi- mained tentative. Consequently MDRC ployer community, essentially giving cally on the effectiveness of mentoring concluded that there were important added support to school-to-work alter- as a youth program intervention — barriers to developing and implement- natives. It highlighted the social costs of P/PV’s Adult/Youth Relationships Pilot ing a large-scale youth apprenticeship ignoring the educational and occupa- Project, as well as Temple University’s system in which a significant part of the tional skill deficits of American young Linking Lifetimes programs, Campus learning was workplace-based. The firm people. A similar theme was evident in Compact’s college-based mentoring suggested building on models such as the Southern Education Foundation’s projects and a program sponsored by academies and Tech Prep, incorporating 1995 report on the work of its Panel on the I Have A Dream Foundation. A se- incentives to encourage employer par- Educational Opportunity and Postsec- ries of reports were produced in 1992, ticipation. ondary Desegregation titled Redeeming including College Students As Mentors In 1996, MDRC produced a report on the American Promise. The panel found for At Risk Youth by J. Tierney and A. its evaluation of the implementation of significant remnants of segregation Branch, and Adult/Youth Relationships career academies, Career Academies: shaping higher education, reducing edu- Pilot Project by J. Greim. These reports Early Implementation Lessons from a cational opportunity for minority stu- revealed the complexity of successful Ten-Site Evaluation, by J. Kemple and J. dents. The report called for an educa- mentoring programs as well as their Rock. The evaluation focused on the tional system from kindergarten through clear benefits. characteristics of academies, student re- four-year colleges that was more stu- cruitment and selection, enrollment and dent-centered and performance-driven, The Manpower Demonstration attrition patterns, and the characteris- and for a more comprehensive, system- Research Corporation (MDRC) tics of students and teachers. Also in wide approach to equal educational op- In the early 1990s, MDRC studied a 1996, MDRC disseminated its evaluation portunity. number of innovative school-to-work of Ohio’s Learning, Earning and Parent- transition programs around the country, ing Program (LEAP). LEAP responded to U.S. Congress: Office of and the role played by employers in two major concerns: the long period Technology Assessment (OTA) teenage mothers remain on public assis- these programs. In 1993, MDRC pro- Another 1995 report is an important tance, and the high dropout rate for duced a report following the study, The resource, Learning to Work: Making the poor teenagers in the educational sys- School-to-Work Transition and Youth Ap- Transition from School to Work. The re- tem. The evaluation covered twelve prenticeship by T. Bailey and D. Merritt. port indicated that work-based learning counties and was reported in a series of The report focused on the youth ap- can help students understand the rele- reports. The final report, Three-Year Im- prenticeship model, the 4-H program, vance of academic studies to their later pacts of Ohio’s Welfare Initiative to Im- cooperative education, career academies lives, assist them in exploring career op- prove School Attendance Among Teenage and Tech Prep. tions, encourage them to acquire desir- Parents by D. Long et al, analyzes able work attitudes and habits, contrib- Interest in youth apprenticeship LEAP’s effect on school completion, em- ute to the development of occupational models grew dramatically in the U.S. in ployment and welfare receipt for a sub- skills, and prepare them for lifelong the 1990s based on an admiration for sample of teens three years following el- learning. OTA explored a range of the German apprenticeship system. The igibility for the program. U.S. Department of Labor created an Of- school-to-work alternatives and provid- fice of Work-Based Learning which The first evaluation report showed ed these insights: launched a number of pilot projects to that LEAP was feasible to operate. The n Workplace learning is most likely to second found that the program prevent- test apprenticeship-type programs, and be effective when planned jointly by ed some in-school teens from dropping foundations such as Jobs for the Future schools and employers, when fo- out and brought some dropouts back supported experimentation with appren- cused not only on occupational skill into the system. The third showed that ticeship approaches. However, the Ger- development but on personal devel- increased school attendance translated man socio-economic environment dif- opment and a broad understanding to a significant increase in school com- fered significantly from American of industry, when coordinated with the academic and occupational in-

Evaluation Forum n Issue 12 n Summer 1997 89 Resources

nn struction of schools, and when stu- Task Force on Education of Young Adoles- dents’ progress is evaluated frequent- Useful References on cents: Turning Points: Preparing Ameri- ly. can Youth for the Twenty-First Century. Youth-Related Issues Washington, DC: Carnegie Council on n The effects of school-to-work sys- Adolescent Development, 1992. tems on student success and work- and the Evaluation of Walker, G. Dilemma and Directions for Youth force productivity will require more Social Policy. Philadelphia, PA: Public/ than a decade to evaluate, though in- Youth Programs Private Ventures, 1994. terim assessments of implementation and student performance can be use- Youth Problems ful in refining systems as they devel- Youth Development op. Abraham, K. G. and S.N. Houseman. Job Se- Center for Youth Development and Policy Re- curity in America: Lessons from Germany. n Commitment and resources from par- search: Preparing Youth for Adulthood: Washington, D.C.: The Brookings Institu- ticipating businesses are imperative Common Ground Between the School-to- tion,1993. regarding planning and implement- Work Youth Development Fields. Wash- Acs, G. and S.H. Danziger. “Educational At- ing a sequence of training, mentoring ington, DC: Academy for Educational De- tainment, Industrial Structure, and Male velopment, 1994. and supervised work experience that Earnings through the 1980s,” in Journal is coordinated with school-based in- Children’s Defense Fund: The State of Ameri- of Human Resources 28, 1993. struction. ca’s Children, 1995. Washington, DC, Aronowitz, S. and W. DiFazio. The Jobless 1995. Future: Sci-Tech and the Dogma of Work. n Employers have been reluctant to Coleman, J. S. (editor) Youth: Transition to Minneapolis, MN: University of Minneso- participate because of inadequate Adulthood. Report of the Panel on Youth. ta Press, 1994. preparation of students for work- Washington, DC: President’s Science Ad- American Policy Forum: Preparing Youth for place assignments, the lack of assis- visory Committee, 1973. the Information Age: A Federal Role for tance in preparing and implementing Commission on Work, Family and Citizen- the 21st Century. Washington, DC: 1996. work-based learning, the costs of ship: The Forgotten Half: Pathways to Avery, R. B. and A. B. Kennickell. “U.S. training and mentoring, the cost of Success for America’s Youth and Young wages, resistance on the part of man- Household Wealth: Changes from 1983 to Families. Washington, DC: William T. 1986,” in Research in Economic Inequali- agement and incumbent employees, Grant Foundation, 1988. ty, Volume 4, edited by E.N. Wolff. and the absence of incentives for in- Edelman, M. and R. Solow. Wasting Ameri- Greenwich, CT: JAI Press, 1993. volvement. ca’s Future. New York, NY: Children’s De- Bailey, T. Education and the Transformation fense Fund, 1994. of Markets and Technology in the Textile Lerner, R. (editor) Early Adolescence: Per- Industry. National Center on Education spectives on Research, Policy, and Inter- and Employment. Technical paper 2. New Editor’s Note vention. Hillsdale, NJ: Lawrence Erlbaum York, NY: Teachers College, Columbia Associates, Inc., 1993. University, 1988. For a provocative and useful analy- National Commission on Children: Beyond Bane, M.J. and D. T. Ellwood. Welfare Reali- sis of youth issues, please see Gary Rhetoric: A New Agenda for Children and ties: From Rhetoric to Reform. Cambridge, Walker’s speech in P/PV’s newsletter Families. Washington, D.C.: U.S. General MA: Harvard University Press, 1994. for fall 1993, “Abandoning the Mechan- Printing Office, 1991. ic’s Model: A New Direction for Youth Bawden, D. L. and F. Skidmore. Rethinking Employment Policy.” Walker’s “develop- National Youth Employment Coalition: To- Employment Policy. Washington, D.C.: ward a National Youth Development Sys- The Urban Institute Press, 1989. mental model” involves 1) developing tem. A report to the Secretary of Labor. Belous, R. S. The Contingent Economy: The the theory and discipline necessary to Washington, DC: 1994. make difficult choices about which ele- Growth of the Temporary, Part-Time and Pittman, K. et al. Youth Employment Prepara- Subcontracted Workforce. Washington, ments of youth programs should be im- tion Through a Youth Development Lens: D.C.: National Planning Association, plemented, 2) investing up front in de- Broad Recommendations for Sustaining 1989. veloping the content to be delivered to Change. Washington, DC: National Youth youth once choices are made, 3) repli- Berryman, S. “Apprenticeship as a Paradigm Employment Coalition and the U.S. De- of Learning,” in Education Through Occu- cating successful elements and pro- partment of Labor, 1994. pations in American High Schools, Vol- grams, 4) continually collecting evi- Portes, A. (editor) The New Second Genera- ume I: Approaches to Integrating Academ- dence of effectiveness, and 5) working tion. New York, NY: Russell Sage Founda- ic and Vocational Education, edited by N. with the private sector to define appro- tion, 1996. Grubb. New York, NY: Teachers College priate roles for employers. Press, 1995. Rutter, M. Changing Youth in a Changing So- ciety: Patterns of Adolescent Development Blank, R. M. and D. Card. “Poverty, Income and Disorder. Cambridge, MA: Harvard Distribution, and Growth: Are They Still University Press, 1980. Connected?” in Brookings Papers on Eco- Secretary’s Commission on Achieving Neces- nomic Activity 2, 1993. sary Skills: What Work Requires of Blinder, A. “The Level and Distribution of Schools. Washington, DC: U.S. Depart- Economic Well-Being,” in The American ment of Labor, 1991. Economy in Transition edited by M. Feld- Takanishi, R. and D. Hamburg. Preparing stein, 1980. Young Adolescents for the 21st Century: ______Hard Heads, Soft Hearts: Tough- Challenges Facing Europe and the United Minded Economics for a Just Society. States. New York, NY: Cambridge Univer- Reading, MA: Addison-Wesley, 1987. sity Press, 1995.

90 Evaluation Forum n Issue 12 n Summer 1997 Resources

Bosworth, B., G. Burtless, and J. Sebelhaus, Goldin, C. and R. A. Margo. “The Great Lampman, Robert. The Share of Top Wealth- “The Decline in Saving: Some Microeco- Compression: The Wage Structure in the Holders in National Wealth. Princeton, nomic Evidence,” in BPEA, 1991. U.S. at Mid-Century,” in Quarterly Jour- NJ: Princeton University Press, 1962. Bounds, J. and R. Freeman. “What Went nal of Economics 107, 1992. Lawrence, R. Z. and M. J. Slaughter. “Inter- Wrong? The Erosion of Relative Earnings Gottschalk, P., S. McLanahan, and G. Sande- national Trade and American Wages in and Employment Among Young Black fur. The Dynamics and Intergenerational the 1980s: Giant Sucking Sound or Small Men in the 1980s,” in Quarterly Journal Transmission of Poverty and Welfare Par- Hiccup?” in Brookings Papers on Econom- of Economics 107, 1992. ticipation. Madison, WI: Institute for Re- ic Activity 2, 1993. search on Poverty, 1992. ______“Changes in the Structure of Wag- Levy, F. Dollars and Dreams: The Changing es in the 1980s: an Evaluation of Alterna- Greenwood, D. T. “Age, Income, and House- American Income Distribution. New York, tive Explanations,” in American Econom- hold Size: Their Relation to Wealth Distri- NY: Russell Sage Foundation, 1987. ic Review 3, 1992. bution in the United States,” in Interna- Lenski, G. E. Power and Privilege. New York, Bragg, D. and J. Layton. “Tech Prep Imple- tional Comparisons of the Distribution of NY: McGraw-Hill, 1966. Household Wealth edited by E.N. Wolff. mentation in the U.S.: The Once and Fu- Lerman, R. I. and H. Pouncy. “The Compel- ture Role of Community Colleges, “ in New York, NY: Oxford University Press, 1987. ling Case for Youth Apprenticeships,” in Community College Review, spring 1995. Public Interest, Fall 1990. Bumpass, L. and J. Sweet. “Children’s Expe- Grubb, N. Learning to Work: the Case for Re- Levy, F, and R. Murnane. “Earnings Levels rience in Single-Parent Families: Implica- integrating Job Training and Education. and Earnings Inequality,” in Journal of tions for Cohabitation and Marital Transi- Kalamazoo, MI: Upjohn Institute for Em- Economic Literature, September 1992. tions,” in Family Planning Perspectives ployment Research, 1996. 21, 1989. Hahn, A. and R. Lerman. What Works in Lipsey, R. and H. Tice (editors) The Mea- Youth Employment Policy? How to Help surement of Saving, Investment and Burtless, G. (editor) A Future of Lousy Jobs? Young Workers from Poor Families. Wash- Wealth. Studies of Income and Wealth, The Changing Structure of U.S. Wages. ington, D.C.: National Planning Associa- Volume 52. Chicago, IL: University of Washington, D.C.: The Brookings Institu- tion, 1985. Chicago Press, 1989. tion, 1990. Heckman, J., R. Roselius, and J. Smith. Litan, R.E., R.Z. Lawrence, and C.L.Schultze Cappelli, P. British Lessons for School-to- (editors) American Living Standards: Work Transition Policy in the U.S. Phila- “U.S. Education and Training Policy: A Re-evaluation of the Underlying Assump- Threats and Challenges. Washington, delphia, PA: National Center on the Edu- D.C.: Center for Economic Progress and cational Quality of the Workforce, 1993. tions Behind the ‘New Consensus’.” Poli- cy paper. Chicago, IL: University of Chi- Employment, The Brookings Institution, Center for Workforce Preparation: New Cen- cago Department of Economics, 1993. 1988. tury Workers: Effective School-to-Work Heckman, J. Assessing Clinton’s Program on Lynch, L. M. “Private-Sector Training and Transition Programs. Washington, DC: the Earnings of Young Workers,” in 1994. Job Training, Workfare, and Education in the Workplace. Cambridge, MA: National American Economic Review, March 1992. Chubb, J. E. and T. M. Moe. Politics, Mar- Bureau of Economic Research, 1993. Magnet, M. The Dream and the Nightmare: kets, and America’s Schools. Washington, The Sixties’ Legacy to the Underclass. D.C.: The Brookings Institution, 1990. Hungerford, T. I. “U.S. Income Mobility in the Seventies and Eighties,” in Review of New York, NY: William Morrow, 1993. Farber, H.S. “The Incidence and Costs of Job Income and Wealth 39, 1993. Martin, J. and A. Murray. “Distributive Injus- Loss: 1982-1991,” in Brookings Papers on Ihlandfeldt, K. Job Accessibility and the Em- tice and Unfair Exchange,” Chapter 7 in Economic Activity, 1993. ployment and School Enrollment of Teen- Equity Theory: Psychological and Socio- Foster, M. “Why Teens Do Not Benefit from agers. Kalamazoo, MI: Upjohn Institute logical Perspectives edited by D.M. Work Experience Programs: Evidence for Employment Research, 1996. Mesick and K.S. Cook. New York,NY: from Brother Comparisons,” in Journal of Praeger, 1983. Jencks, C. and P.E. Peterson (editors) The Policy Analysis and Management, sum- Mead, L. The New Politics of Poverty: The mer 1995. Urban Underclass. Washington, D.C.: The Brookings Institution, 1991. Working Poor in America. New York, NY: Freeman, R.B. “How Much Has De-Unioniza- Basic Books, 1992. tion Contributed to the Rise in Male Juhn, C., K.Murphy and R. Topel. Why Has National Governors Association: The Em- Earnings Inequality?” in Uneven the Natural Rate of Unemployment In- ployer Connection: State Strategies for Tides edited by S.H. Danziger and P. creased Over Time? Brookings Papers on Building School-to-Work Partnerships. Gottschalk. New York, Russell Sage Foun- Economic Activity. Washington, D.C.: Washington, DC: 1995. dation, 1993. The Brookings Institution, 1991. Karoly, L.A. The Trend in Inequality Among ______: European School-to-Work Sys- Freeman, R.B. and H. Holzer, The Deteriora- Families and Workers in the United tems: A View from the American States. tion of Employment and Earnings Oppor- States: A Twenty-Five Year Perspective. Washington, DC: 1995. tunities for Less Educated Young Ameri- Santa Monica, CA: Rand, 1992. cans. Washington, D.C.: National Bureau Palmer, J. L. Creating Jobs: Public Employ- of Economic Research, August 1991. Katz, L. F. and K.M. Murphy. “Changes in ment Programs and Wage Subsidies. Relative Wages, 1963-1987: Supply and Washington, D.C.: The Brookings Institu- General Accounting Office: Training Strate- tion, 1978. gies: Preparing Noncollege Youth for Em- Demand Factors,” in Quarterly Journal of ployment in the U.S. and Foreign Coun- Economics 107, 1992. Pechman, J. A. Who Paid the Taxes, 1966- tries. Washington, DC: May, 1990. Krueger, A. B. How Computers Have Changed 1985? Washington, D.C.: The Brookings Institution, 1985. ______: Transition from School to Work: the Wage Structure: Evidence from Micro- Linking Education and Worksite data, 1984-1989. Princeton, NJ: Princeton Porter, M.E. “Capital Disadvantage: Ameri- Training.Washington, DC: August, 1991. University, 1991. ca’s Failing Capital Investment System,” in Harvard Business Review, Volume 70, September-October, 1992.

Evaluation Forum n Issue 12 n Summer 1997 91 Resources

Schultze, C. L. and H. J. Aaron (editors) Set- Challenges to Agencies, Programs and So- sues in the Evaluation of Manpower Pro- ting Domestic Priorities: What Can Gov- cial Policy. Washington, D.C.: The Urban grams,” in Industrial and Labor Relations ernment Do? Washington, D.C.: The Institute Press, 1994. Review, 1972. Brookings Institution, 1992. Moss, P. and C. Tilly. Why Black Men Are Bureau of International Labor Affairs: For- Solow, R. M. “On Theories of Unemploy- Doing Worse in the Labor Market: A Re- eign Economic Research Division: The ment,” in American Economic Review, view of Supply-Side and Demand-Side Ex- Wage Structure by Occupation: Skill Level March 1980. planations. New York, NY: Social Science and Skill, and The Impact of Technology, Research Council, 1992. Trade and Outsourcing on Employment Thurow, L.C. Generating Inequality: Mecha- and Labor Composition. Washington, DC: nisms of Distribution in the U.S. Econo- Ogbu, J. U. Minority Education and Caste: U.S. Department of Labor, 1994. my. New York, NY: Basic Books, 1975. the American System in Cross-Cultural U.S. General Accounting Office: Training Perspective. New York, NY: Academic Bureau of Labor Statistics: Technological Strategies: Preparing Noncollege Youth for Press, 1978. Change and On-the-Job Training of Young Employment in the U.S. and Foreign Sum, A. and N. Fog. The Changing Economic Workers: The Intergenerational Conse- Countries. Washington, D.C.: The U.S. Fortunes of Young Black Men in the New quences of Family Structure on Wages and Congress, May 1990. American Economy of the 1980s. Boston, Educational Attainment. Washington, DC: U.S. Department of Labor, 1996. ______: Transition from School to Work. MA: Northeastern University, 1989. Washington, D.C.: The U.S. Congress, U.S. Department of Education: America’s Burtless, G. and L. Orr. “Are Classical Exper- 1991. High School Sophomores: A Ten-Year iments Needed for Manpower Policy?” in Comparison. Washington, D.C.: National Journal of Human Resources 21, 1986. Vedder, R. and L. Gallaway. Out of Work: Un- Center for Education Statistics, 1993. Campbell, D.T. and J.C. Stanley. Experimen- employment and Government in Twenti- tal and Quasi-Experimental Designs for eth Century America. New York, NY: Williamson, J. and P. Lindert. American Ine- Research. Chicago, IL: Rand McNally, Holmes and Meier, 1993. quality: A Macroeconomic History. New 1966. Walster, E.G., W. Walster, and E. Berscheid. York, NY: Academic Press, 1980. Equity: Theory and Research. Boston, Wolff, E. N. “Changing Inequality of Cave, G., H. Bos, F. Doolittle and C. Tous- MA: Allyn and Bacon, 1978. Wealth,” in American Economic Review saint. JOBSTART: Final Report on a Pro- 82, 1992. gram for School Dropouts. New York, NY: Manpower Demonstration Research Cor- Minority Youth poration, 1993. Evaluation of Youth Programs Blalock, H.M.,Jr. Toward a Theory of Minori- Cave, G. and J. Quint. Career Beginnings Im- ty Group Relations. New York, NY: John Ashenfelter, O. “Estimating the Effect of pact Evaluation: Findings from a Program Wiley, 1967. Training Programs on Earnings,” in Re- for Disadvantaged High School Students. view of Economics and Statistics, 60, Bobo, L. and R. Smith. “Antipoverty Policy, New York, NY: Manpower Demonstration Affirmative Action,and Racial Attitudes,” 1978. Research Corporation, 1990. in Confronting Poverty: Prescription for Bailey, T and D. Merritt. The School-to-Work Connell, J., L. Aber and G. Walker. “How So Change edited by S.H. Danziger, G. Sand- Transition and Youth Apprenticeship: Les- Urban Communities Affect Youth? Using efur, and D. H. Weinberg. Cambridge, sons from the U.S. Experience. Manpower Social Science Research to Inform the De- MA: Harvard University Press, 1994. Demonstration Research Corporation, sign and Evaluation of Comprehensive 1993. Cutler, D. M. and L.F. Katz. “Macroeconomic Community Initiatives,” in New Ap- Performance and the Disadvantaged” in Barnow, B.S., G. Cain, and A. Goldberger. proaches to Evaluating Comprehensive Brookings Papers on Economic Activity 2, “Issues in the Analysis of Selectivity Community Initiatives: Concepts, Meth- 1991. Bias,” in Evaluation Studies 5, 1980. ods, and Context edited by J. Connell, C. Weiss et al. Queenstown, MD: Aspen In- Fisher, S. (editor) Power and the Black Com- Bassie, L. “Estimating the Effect of Training stitute, 1995. munity. New York, NY: Random House, Programs with Non-Random Selection,” 1970. in Review of Economics and Statistics 66, Cook, T.D. and D.T. Campbell. Quasi-Experi- 1984. mentation: Design and Analysis Issues Hauser, R. and H. Phang. Trends in High for Field Settings. Chicago, IL: Rand Mc- School Dropout Among White, Black, and Bishop, J. H. “Toward More Valid Evalua- Nally, 1979. Hispanic Youth, 1973-1989. Madison, WI: tions of Training Programs Serving the Institute for Research on Poverty, 1993. Disadvantaged,” in Journal of Policy Corson, W. and M. Silverberg. “Learning at Analysis and Management 8, 1989. the Work Site,” in School-to-Work/Youth Ihlanfeldt, K. and D. Sjoquist. “Job Accessi- Apprenticeship Demonstration: Prelimi- bility and Racial Differences in Youth Em- Blalock, H.M.,Jr. Causal Inferences in Nonex- nary Findings published by the U. S. De- ployment Rates,” in American Economic perimental Research. Chapel Hill, NC: partment of Labor, 1994. Review, March 1990. University of North Carolina Press, 1964. Dickinson, K.P. et al. Evaluation of the Ef- Jaynes, G.D. and R.M. Williams, Jr. (editors) ______Conceptualization and Measure- fects of Performance Standards on Clients, A Common Destiny: Blacks and Ameri- ment in the Social Sciences. Beverly Hills, Services, and Costs. Washington, D.C.: can Society. Washington, D.C.: National CA: Sage, 1982. National Commission for Employment Research Council, 1989. Bloom, H.S. “Estimating the Effect of Job Policy, 1988. Kenney, G. and D. Wissoker. “An Analysis of Training Programs Using Longitudinal Fishman, M. and D. Weinberg. “The Role of the Correlates of Discrimination Faced by Data,” in Journal of Human Resources 19, Evaluation in State Welfare Reform Young Hispanic Jobseekers,” in American 1984. ‘Waiver’ Demonstrations,” in Evaluating Economic Review, June 1994. Bloom, H., L. Orr, F. Doolittle, J.Hotz and B. Welfare and Training Programs edited by Melendez, D. “Understanding Latino Pover- Barnow. Design of the National JTPA C. Manski and I. Garfinkel. Cambridge, ty,” in Hispanic Journal of Behavioral Sci- Study. Bethesda, MD: Abt Associates and MA: Harvard University Press, 1991. ences 14, 1992. MDRC, 1988. Fraker, T. and R. Maynard. “The Adequacy of Mincy, R. B. Nurturing Young Black Males: Borus, M. and C. Buntz. “Problems and Is- Comparison Group Designs for Evalua- tions of Employment-Related Programs,”

92 Evaluation Forum n Issue 12 n Summer 1997 Resources

in Journal of Human Resources 22, 1987. nomic Review 76, 1986. Smith, T. et al. Launching AmeriCorps: First- Furano, K. et al. Big Brothers/Big Sisters: A Light, A. Transitions from School to Work: A Year Implementation. Philadelphia, PA: Study of Program Practices. Philadelphia, Survey of Research Using the National Public/Private Ventures, 1995. PA: Public/Private Ventures, 1983. Longitudinal Surveys. Washington, DC: Stern, D. et al. “Benefits and Costs of Drop- Bureau of Labor Statistics, U.S. Depart- out Prevention in a High School Program Garfinkel, I., C. Manski and C. Michalopou- ment of Labor, 1994. Combining Academic and Vocational Ed- los. “Are Micro-Experiments Always ucation: Third Year Results,” in Educa- Best?” in Evaluating Welfare and Training Long, D., J. Gueron et al. Three Year Impacts tional Evaluation and Policy Analysis 11, Programs, edited by C. Manski and I. of Ohio’s Welfare Imitative to Improve 1989. Garfinkel. Cambridge, MA: Harvard Uni- School Attendance Among Teenage Par- versity Press, 1991. ents. New York, NY: Manpower Demon- Stern, D. et al. Research on School-to-Work Geronimus, A. and S. Korenman. “The So- stration Research Corporation, 1996. Transition Programs in the United States. cioeconomic Costs of Teenage Childbear- Mathematica Policy Research, Inc.: School-to- Berkeley, CA: National Center for Re- ing: Evidence and Interpretation,” in De- Work Implementation Progress: The State search in Vocational Education, Universi- mography, May 1993. Perspective in Early 1996. Princeton, NJ: ty of California, 1994. Grossman, J. and C. Sipe. Summer Training 1996. Stern, D. et al. Career Academies: Partner- and Education Program (STEP): Report Moffitt, R. “Symposium on the Econometric ships for Reconstructing American High on Long-Term Impacts. Philadelphia, PA: Evaluation of Manpower Training Pro- Schools. San Francisco, CA: Jossey-Bass, Public/Private Ventures, 1992. grams,” in Journal of Human Resources 1992. 22, 1987. Stromsdorfer, E., H. Bloom et al. Recommen- Hahn, A. et al. Evaluation of the Quantum dations of the Job Training Longitudinal Opportunities Program: Did the Program ______Final Report of the Panel on Survey Research Advisory Panel. Washing- Work? Center for Human Resources, Manpower Training Evaluation -- The Use ton, D.C.: U.S. Department of Labor, Heller Graduate School. Waltham, MA: of Social Security Earning Data for As- 1985. Brandeis University, 1994. sessing the Impact of Manpower Training Hausman, J. and D. Wise (editors) Social Ex- Programs. Washington, D.C.: National Tierney, J. and J. Grossman. Making a Dif- perimentation. Chicago, IL: University of Academy of Sciences, 1974. ference: An Impact Study of Big Brothers/ Chicago Press, 1985. Namboodiri, N.K., L.F. Carter, and H.M. Bla- Big Sisters. Philadelphia, PA: Public/Pri- vate Ventures, 1988. Heckman, J. “Randomization and Social Pol- lock. Applied Multivariate Analysis and icy Evaluation,” in Evaluating Welfare Experimental Designs. New York, NY: U.S. General Accounting Office: Job Corps: and Training Programs edited by C. Man- McGraw-Hill, 1975. High Costs and Mixed Results Raise Ques- ski and I. Garfinkel. Cambridge, MA: Office of Policy and Research, Employment tions about Program’s Effectiveness. Harvard University Press, 1991. and Training Administration: The School- Washington, DC: June, 1995. ______”Choosing Among Alternative to-Work /Youth Apprenticeship Demon- Weinbaum, S. and F. Wirmusky. Building on Non-Experimental Methods for Estimat- stration: Preliminary Findings. Washing- Best Practices in Youth Employment: ing the Impact of Social Programs,” in ton, DC: U.S. Department of Labor, 1996. What Works, How Do We Know, How Do Journal of the American Statistical Asso- Orr, M.T. Evaluating School-to-Work Transi- We Sustain and Replicate Them? Wash- ciation 84, 1989. tion. Prepared for National Institute for ington, DC: Academy for Educational De- Work and Learning. Washington, DC: velopment, 1994. Hoachlander. G. “Industry-Based Education: Academy for Educational Development, Wellington, A. “Effects of the Minimum A New Approach for School-to-Work 1995. Wage on the Employment Status of Transition,” in School-to-Work: What Youths,” in Journal of Human Resources, Does Research Say About It? published by Osterman, P. and M. Iannozzi. Youth Appren- winter 1991. the U. S. Department of Education, 1994. ticeships and School-to-Work Transition: Hollenbeck, K. Postsecondary Education As Current Knowledge and Legislative Strate- Triage: Returns to Academic and Techni- gy. Philadelphia, PA: University of Penn- International References cal Programs. Kalamazoo, MI: W.E. Up- sylvania’s National Center on the Educa- Braun, C. et al. Programme for Research and john Institute for Employment Re- tional Quality of the Workforce, 1993. Actions on the Development of the Labour search,1992. Pauly, E., H. Kopp, and J. Haimson. Home- Market: Youth Unemployment, Delin- quency and the Inner City: Final Report. Jobs for the Future: Promising Practices: A grown Lessons: Innovative Programs Brussels, Belgium: Commission of the Study of Ten School-to-Career Programs. Linking School and Work. San Francisco, European Communities, 1996. Boston, MA: 1995. CA: Jossey-Bass, 1995. Plotnick, R.”The Effects of Attitudes on Circle for Youth Research Cooperation in Eu- Kane, T. and C. Rouse. Labor Market Returns Teenage Premarital Pregnancy and Its rope: European Yearbook on Youth Policy to Two-and-Four-Year Colleges. Cam- Resolution,” in American Sociological Re- and Research, New York, NY: Walter de bridge, MA: National Bureau of Economic view, December 1992. Gruyter, 1995. Research, 1993. Polit, D. et al. The Challenge of Serving Teen- Gazso, F. and V. Shubkin (editors) Working Kish, L. Statistical Design for Research. New Youth — Aspirations and Occupational York, NY: John Wiley, 1987. age Mothers: Lessons from Project Redirec- tion. New York, NY: Manpower Demon- Careers. Budapest, Hungary, 1980. Klerman, J. and L. Karoly. “ Young Men and stration Research Corporation, 1988. Hanratty, M. and R. Blank. “Down and Out the Transition to Stable Employment, “ in in North America: Recent Trends in Pov- Monthly Labor Review, August 1994. Quint, J. et al. New Chance: Interim Findings on a Comprehensive Program for Disad- erty Rates in the U.S. and Canada,” in Krueger, A. and C. Rouse. New Evidence on vantaged Young Mothers and Their Chil- Quarterly Journal of Economics 107, Workplace Education. Cambridge, MA: dren. New York, NY: Manpower Demon- 1992. National Bureau of Economic Research, stration Research Corporation, 1994. Hartman, J. “Research Project: Integration of 1994. Riccio, J. et al. GAIN: Benefits, Costs, and Youth into Society,” in International So- Lalonde, R. “Evaluating the Econometric Three-Year Impacts of a Welfare-to-Work ciological Association’s International Bul- Evaluations of Training Programs with Program. New York, NY: Manpower Dem- letin for Youth Research, volume 1,1983. Experimental Data,” in American Eco- onstration Research Corporation, 1994. Jantti, M. and S. Danziger. “Child Poverty in

Evaluation Forum n Issue 12 n Summer 1997 93 Resources

Sweden and the U.S.: The Effect of Social Employment Projects for Unemployed Organization for Economic Cooperation and Transfers and Parental Labor Force Partic- Young People in the European Communi- Development: Evaluating Labour Market ipation,” in Industrial and Labor Rela- ty. Brussels, Belgium: European Centre and Social Programmes. Paris, France, tions Review 48, 1994. for the Development of Vocational Train- 1991. Kdlitz, V. Youth Unemployment and Voca- ing, 1996. ______: Apprenticeship: Which Way For- tional Training: The Material and Social Kuczynski, J., S. Eisenstadt, B. Ly and L. ward? Paris, France: 1995. Standing of Young People During Transi- Sharkar. Perspectives on Contemporary Rademacker, H.and P. Evans. “Transition tion from School to Work. Brussels, Bel- Youth. New York, NY: United Nations from School to Work,” in Our Children at gium: Commission of the European Com- University, 1988. Risk. Paris, France: OECD, 1995. munities, 1995. Murray, C. and A. Dawson: “Five Thousand Social Europe Series: Youth Pay and Employ- Kerr, K. Youth Unemployment in Canada. Ot- Adolescents: Their Characteristics, Atti- ers’ Recruitment Practices for Young Peo- tawa, Canada: Library of Parliament, tudes and Attainments.” Paper presented ple. Employment, Social Affairs and Edu- 1991. at the European Centre for Social Welfare, cation Division. Brussels, Belgium: Ketter, P. Training for Everyone: A Guide to Training and Research, Talberg, Sweden: European Commission, 1996. the Planning of Innovative Training and Expert Meeting on Youth Integration through Participation. 1983. UNESCO: Youth in the 1980s. Paris, 1981.

94 Evaluation Forum n Issue 12 n Summer 1997 Material contained in this publica- Evaluation Forum is a national jour- Editor and Reviewer tion is in the public domain and may nal for program professionals in re- Ann Bonar Blalock be reproduced, fully or partiallly, lated human service fields: basic Admiralty Inlet Consulting without permission of the Federal education, vocational education, em- Hansville, Washington Government. Source credit is re- ployment and training, welfare, labor Guest Reviewer quired only to reproduce any copy- market and labor force change, and Ernst W. Stromsdorfer righted material contained herein. economic development. The pur- Department of Economics pose of the journal is to stimulate in- Washington State University This material will be made available terest in policy analysis, policy re- Pullman, Washington to sensory impaired individuals search and program evaluation as Graphics and Technical Services upon request. useful tools for policymaking, plan- Kathleen Shankman Voice phone: 202-219-5782 ning and the improvement of social Center for Governmental Studies TDD* phone: 1-800-328-2577 programs. It is understood that the Northern Illinois University ideas and opinions expressed in the DeKalb, Illinois *Telecommunications Device for the content of the journal are those of Deaf. the respective writers and do not necessarily represent those of the funder or those of the contributing public interest organizations. Sponsor and Funder Office of Policy and Research Employment and Training Administration U.S. Department of Labor Contributors to Journal Distribution National Governors’ Association Washington, D.C. National Association of Counties Washington, D.C. National Alliance of Business Washington, D.C. History of the Journal field. Its major objective is to com- Reader Participation Evaluation Forum was developed municate information on policy is- The content of the new version of as part of a five-year national dem- sues and research activities at the the journal will cover the same kinds onstration project, The JTPA Evalu- national, state and local level which of materials as in the past: commen- ation Design Project, funded be- can inform judgments about the de- taries, policy analyses; program re- tween 1986 and 1991 by the National sign and effectiveness of various views; reports on evaluation plan- Commission for Employment Policy, employment and training policies ning, reports on analyses of the U.S. Department of Labor, the and strategies. Eight theme-oriented monitoring data; research reports on IBM Corporation and The Ford Foun- issues of the journal were published program evaluations or policy re- dation. The purpose of the project as part of the demonstration project. search; books and articles on policy was to increase the interest, knowl- The New Evaluation Forum issues, programs and research; edge and sophistication of employ- The new Evaluation Forum, of and other materials related to evalu- ment and training professionals at which this is the fourth issue, is be- ation that would be of interest to the state and local level regarding ing sponsored and funded by the practitioners. program evaluation: its scientific Office of Planning and Research, principles and methods, its practical Employment and Training Adminis- applications in better understanding tration, U.S. Department of Labor. program implementation and impact, Its purpose is to disseminate useful and its utility as a practical tool for information on evaluation issues, ac- adjusting policies and improving pro- tivities and results to those respon- grams. The project’s series of sible for administering, planning, evaluation guides can be obtained managing and overseeing employ- through the ERIC system in public li- ment and training programs across braries, or in book form from the W. the United States. Again, each issue E. Upjohn Institute for Employment of the journal is theme-oriented, and Research. follows the same organizational for- Evaluation Forum is a national mat as in the past. However, each journal growing out of the five-year issue has featured reviews by the project and developed specifically editor and guest reviewer rather than for state and local program practitio- articles written by practitioners and ners in the employment and training researchers.