Proceedings of the Conference on Legislative Oversight October 13-15, 1985

_ Tbe Joint Legislative ·re Audit and Review Commission Ill! of tbe Virginia General Assembly

) I The Honorable Members June 1, 1986 of the Virginia General Assembly State Capitol Richmond, Virginia

My Dear Colleagues:

When the Legislative Program Review and Evaluation Act was designed and enacted in 1978, its creators felt strongly that the Act itself should be subject to review after an appropriate period. A provision of the Act stipulated that "in 1985 a Conference on Legislative Oversight will be held by the Joint Legislative Audit and Review Commission to assess and evaluate the accomplishments of this act." -- To comply with this requirement, JLARC sponsored the Conference in October 1985. The event was co-hosted by Speaker of the House A. L. Philpott, Delegate L. Cleaves Manning, and myself. The Conference provided a useful forum for surveying the last seven years of JLARC's work, placing that work in the larger context of legislative oversight across the country, and proposing improvements to the Evaluation Act based on actual experience with it in the legislature. Subsequent to the Conference, a number of significant changes occurred. The legislature approved most of the changes to the Evaluation Act that grew out of the Conference. JLARC's staff director of 12 years, Ray D. Pethtel, accepted the Governor's appointment as Virginia's new Commissioner of Highways and Trans­ portation. The former Deputy Director, Philip A. Leone, was appointed to the directorship and confirmed by the 1986 Session of the General Assembly. Two new members, Delegate Alson H. Smith, Jr., and Delegate William T. Wilson, were appointed to the Commission. I was elected Chairman and Delegate Theodore V. Morrison was elected Vice-Chairman for the biennium. The publication of these Conference proceedings, which was also required under the Evaluation Act, is an opportnnity for me to thank those who have contributed so much to the success of legislative oversight in the Commonwealth. My thanks go to all my legislative colleagues who sponsored and participated in the conference, served on the Commission and its many subcommittees over the years, and supported JLARC's work on the floor of the House and Senate. And to my predecessor as Chairman, Cleaves Manning, I know I speak for the entire Commission when I express our gratitude for his leadership and commitment.

Respectfully,

Edward E. Willey Chairman JLARC Members of the Joint Legislative Audit and Review Commission

Chairman Senator Edward E. Willey

Vice Chairman Delegate Theodore V. Morrison, Jr.

Senator Hunter B. Andrews Delegate Robert B. Ball, Sr. Senator Peter K. Babalas Senator John C. Buchanan Delegate Vincent F. Callahan, Jr. Delegate Lacey E. Putney Delegate Ford C. Quillen Delegate Alson H. Smith, Jr. Delegate William T. Wilson Mr. Walter S. Kucharski, Auditor of Public Accounts Preface

On October 13-15, 1985, the Joint Legislative Audit and Review Commission (JLARC) held a Conference on Legislative Oversight at the Hotel John Marshall in Richmond. The Conference was required under provisions of Chapter 388 of the 1978 Acts of Assembly, which established the Legislative Program Review and Evaluation Act. Speaker A. L. Philpott, Senator Edward E. Willey, and Delegate L. Cleaves Manning co-hosted the event. In addition to members of the General Assembly and the staff of JLARC, legislators and oversight staff from around the country were invited to the Conference. More than 125 persons, representing 32 states and Canada, took part in the forum. Panelists and speakers discussed a number of important issues and trends in legislative oversight. The third day was devoted to a regional meeting of the Legislative Program Evaluation Section of the National Conference of State Legislatures. Thus, the purpose of the conference was two-fold. First, it provided interested members of the General Assembly with an opportunity to examine the accomplishments of the Evaluation Act, as intended by provisions of the Act itself. Second, it provided a international forum for addressing oversight issues and sharing insights. This document serves as the proceedings of the Oversight Conference. As such, it provides a broad range of information that should appeal to the wide spectrum of participants in legislative oversight: oversight commission members, other legislators, management and research staff working for oversight groups, and others in the private and academic sectors who are interested in this aspect of government. It should be noted that the panel discussions and invited remarks that comprised much of the Conference varied not ouly in content, but also in format. Some of the presentations included in these proceedings are formal papers; others are transcripts of informal remarks. The specific Conference recommendations regarding Virginia's Evaluation Act are found in the section entitled "Directions for Oversight in Virginia." A more detailed discussion of these recommendations was presented in an earlier publication, JLARC's 1985 Report to the General Assembly. The amended form of the Act which was passed by the 1986 General Assembly is included as an appendix to the proceedings, along with the complete agenda of the Conference. Finally, I would like to express my sincere appreciation to all of the speakers and panel members for contributing to the success of the JLARC Conference on Legislative Oversight.

Philip A. Leone Director Table of Contents Page Status of Oversight in the States Dr. Alan Rosenthal, Eagleton Institute of Politics, Rutgers University 1 Status of Sunset in the States Richard Jones, Program Director, Legislative Management, National Conference of State Legislatures 5 Need for Legislative Oversight and Legislator Expectations of Evaluation Staff Wayne Fawbush, Representative, Oregon 9 Mordecai Lee, Senator, Wisconsin , ....•...... 13 Herbert F. Morlan, Representative, Florida ...... •...... l 7 Oversight in the Congress and the U.S. General Accounting Office Eleanor Chellmsky, Director, Program Evaluation and Methodology Division, U.s. General Accounting Office 23 Ingredients for an Effective Oversight Function Dale Cattanach, State Auditor, W isconsi n...... •...... •.....•3 1 Georlle L. Schroeder, Executive Director, South Carolina Legislative Audit Council ...••...... •..34 WIlliam Thomson, Director, Performance Audit Division, Office of the Auditor General, Arizona 37 John W. Turcotte, Director, fOint Legislative Committee on Performance Evaluation and Expenditure Review, Mississippi ..., ...... •...•...... •.....•41 Directions for Oversight in Virginia Ray D. Pethtel, Director, foint Legislative Audit and Review Commission, Virginia ...... •.•..44 L. Cleaves Manning, Delegate and Chairman of the foint Legislative Audit and Review Commission, Virginia•...... •...... •...... •...... SO Professional and Ethical Concerns in Legislative Program Evaluation Jacques Goyer, Principal, Office of the Auditor General of Canada 54 Dr. Lelllh Grosenlck, Virginia Commonwealth University 61 Dr. Jack Rabin, Rider College 65 Design and Methodology - How Do Organizations Do It? William Thomson, Director, Performance Audit Division, Office of the Auditor General, Arizona 71

Bernie Gelzer, I Director, Legislative Commission on Expenditure Review, New York .....•.•....075 Dr. Gary Henry, foint Legislative Audit and Review Commission, Virginia 78 Appendixes Legislative Program Review and Evaluation Act as amended by the 1986 General Assembly 83 Conference program, 8 5 Status of Legislative Oversight in the States

As a student of state legislatures, I have watched legislative oversight take root in representative assemblies throughout the nation. Nowhere has it taken firmer, healthier root than here in Virginia. For me this "Conference on Legislative Oversight" is a celebration of JLARC's achievements. The function of performance auditing, program evaluation, Dr. Alan Rosenthal or program review -- it's called different things in different places -- is the major form of legislative oversight. It has been institutionalized in Virginia. By "insti­ Eagleton Institute of Politics tutionalized," I do not mean that those who Rutgers University engage in the enterprise are like inmates in a mental home! I mean rather that =~...... -=""" oversight has become a regular and vital port of the legislative process.

The Development of Oversight The struggle has not been an easy one, for legislatures do not take naturally to oversight. Legislators delight in the business of lawmaking. They like sponsoring bills, pushing them through, and claiming credit if they are enacted. They take to all this like ducks take to water. Legislators also have little trouble with their job of servicing constitutents. They appreciate the benefits from communicating to folks in the district, helping them with their problems, holding their hands, and intervening when they run afoul of the state's impersonal bureaucratic apparatus. At least a number of legislators are also attracted to budget review and appropriations, another major legislative function. In this arena there is the lure of real power, in having a say on how the stste's resources are allocated among competing needs and demands -- and also among districts. Not many legislators, however, make themselves available for the arduous tasks Ststus of Legislative Oversight in the Statss

of reviewing agency performance and A tradition, an ethos of fiscal program effects or are willing to adopt the responsibility -- or fiscal conservatism -- in tough and critical posture required by the legislature helps. Such a tradition in oversight. Few people want to confront part accounts for the effectiveness of others and tell them that they should be JLARC in Virginia. It is important in doing a better job or that what they're Colorado, too. The lack of such a tradition, about is not really working. I suspect, makes it more difficult, although My involvement with Virginia goes not impossible, for the folks who want to do back a ways. I was invited to visit when oversight in Minnesota and New York. JLARC was only a glimmer in the eye of the The balance of power between the General Assembly. In 1972 a special legislative and executive branches also committee had been appointed to look into counts. A strong, independent legislature, the possibility of an oversight process, and such as Colorado's or Florida's, makes for a within a year JLARC was established. more effective enterprise. A more It was not the first such agency in the submissive legislature, such as Connect­ country. Similar operations had already icut's has tended to be over the years, been started in Hawaii, New York, and makes it difficult to follow through on Connecticut. But in the 12 years since its criticism of administrative performance -­ establishment, JLARC has become the especially if the Governor is unhappy with foremost; today it stands as a model for the such criticism. Here, too, Virginia measures rest of the nation. This is not to say that up. Its legislature is an independent one, other states have not made progress of their and has been at least for the last decade or own; they have. Performance auditing and so, if not in more distant history. program evaluation agencies are now Partisan control of state government operating in about two-thirds of the states. can also make a difference. There is more They are hard at work, and their efforts incentive, in competitive party states in make a considerable difference to the particular, for the legislature to be critical conduct of state government. I doubt, of the executive branch when control of however that any other enterprise has been government is divided. Indeed, the as effective and as successful as JLARC. establishment of a number of legislative Why is this so? What accounts for audit or evaluation agencies occurred when JLARC's success? The credit belongs to one party had the office of governor and the JLARC's staff, the General Assembly, and other party had the legislature. This is what to the Commonwealth itself. happened in Connecticut, Florida, Kentucky, Permit me to comment on the factors and Virginia. But does this mean that or conditions that promote legislative JLARC has been more diligent in pursuit of oversight and to assess the extent to which agencies under Republican Governors Mills these factors or conditions are found in Godwin and John Dalton than of agencies Virginia. I'll mention only those I consider under Democratic Governor Charles Robb? to be the most significant ones, leaving a I doubt it. Partisan politics doesn't number -- and particularly those associated influence legislative oversight in Virginia, with the conduct of performance audits and and JLARC is tough -- but fair -­ program evaluations -- to legislators and regardless of the Governor's party. staff on other conference panels. Leadership Commitment The Political Environment The second major factor is the The first factor has to do with the commitment to the oversight enterprise of political environment. Certain environ­ the legislature's leadership. ments are more conducive to legislative Legislative leaders are under oversight than others. extraordinary pressure today. The priorities

Dr. Alsn ROBBIlthsl 2 Statua of Lagis/ativa Ovarsight in tha Statas of most of them do not include oversight. supportive of the enterprise but have also That is because leaders are up to their ears called on JLARC for particular studies. The in politics, trying to win or hold onto leadership of the Virginia General Assembly legislative majorities. They are naturally is committed today, and it bas been involved in policy, and they are forced to committed from the very beginning. deal with the pressing matters of management and administration. There is Committee Membership more on their menu than they can possibly The third major factor is the oversight swallow, and much more than they could committee or commission itself. This ever digest. No wonder that many leaders factor may be the most critical of all. have little time for oversight! A number of standing committees will But leaders do not need to devote an make use of oversight studies, and executive inordinate amount of time to oversight. agencies will take heed of oversight What's required is their concern and recommendations, but there should be a support, on a continuing basis. They must principal and immediate client. That client, signal to members that oversight is a of course, is the committee or commission significant function of the legislature and an of legislators that directs the oversight important responsibility of legislators. staff. JLARC is such a body. Leaders can do this first by The composition and structure of the encouraging able and respected members to committee or commission can vary. There serve on the oversight committee or is no one way to skin a cat. The only commission, and then by appointing the best requirement is that the body have people and not the ones who are left over reputation and prestige in the legislature -­ when the major committees are tilled. that is, some clout. Colorado's Legislative Leaders can do this also by serving on the Audit Committee has it. Connecticut's committee or commission themselves, and Program Review and Investigations not only by serving but by actually Committee used to have it, but bas it no participating. In some states leaders serve, longer. JLARC has as much clout as anyone but only nominally; they rarely attend might wish, with the President Pro Tem and meetings and send staff surrogates in their Majority Leader of the Senate, the place. Chairman of House Appropriations, and In some states the commitment of other influential legislators as members. legislative leadership is questionable. The continuity of membership makes a Mississippi's Speaker of the House bas never tremendous difference. I was impressed been comfortable with the Committee on several years ago while attending a JLARC Performance Evaluation and Expenditure meeting. Members, in reviewing a report Review. New York's leaders have been presented to them by staff, recalled related involved in so much that they had little left studies and events that had occurred years over for oversight. Connecticut's leaders, before. They remembered. They had who were heavily engaged at the outset, memories as individuals and, thus, the lately have lost interest, I am sorry to say. Commission had a collective memory. In other states, like Virginia, several During a period when turnover in leaders not only serve but they truly lead legislatures and on committees has been the committee or commission, as have rather high, JLARC has been extraordinary Senator Edward E. Willey, the President Pro as far as continuity is concerned. Since its Tem, Senator Hunter B. Andrews, the establishment in 1973 only 17 members have Majority Leader, and Delegate Richard M. filled the 11 legislator positions on the Bagley, the Chairman of the House Commission. If I'm not mistaken, Senator Appropriations Committee. Other Virginia Willey and Delegates Cleaves Manning, Ford leaders, especially Speakers John W. Cooke, Quillen, Lacey Putney, and Ted Morrison and A.L. Philpott, not only have been have been members since the very

Dr. A/an Rosantha/ 3 Status of Lsgislativs Ovsrsight in tha StBfBs beginning. Senator Andrews and Delegate political environment. They know how to Bagley came on shortly thereafter. What a respond to the legislature's needs without difference continuity makes, particularly sacrificing their professional integrity. when the members are committed and They know where to go and where not to go, talented to begin with! and just how far to go. Pethtel and Leone Leadership by the chairman also are by no means the only such staff leaders counts, and counts heavily on committees in this business, but they are among the very and commissions across the country. best. Usually the chairmanship rotates -- the senate holds it one year, the house the next; What Virginia Demonstrates the majority has it for a session, and then When it comes to legislative oversight the minority may take over. The -- that is, performance auditing and chairmanship rotates in Virginia so that five program evaluation -- Virginia stands as a individuals have guided JLARC since its model for all the states. JLARC is probably creation. They have given the commission in a class by itself. It is blessed. excellent leadership. There is no JLARC is blessed because it operates legislature, in my opinion, where leadership in a political environment that is conducive and membership of the oversight group have to legislative oversight. It is blessed been as impressive over an extended period because the General Assembly's leadership of time as in Virginia. Leaders and has been so supportive from the very members deserve much of the credit for beginning. It is blessed because its members JLARC's achievements. care, continue, and contribute mightily to the enterprise. It is blessed because its Professional Staff staff is absolutely first-rate. The fourth major factor is professional Not every legislature is in as enviable staff. I must say that the staffmg of a position. Nor can every state emulate the legislative performance auditing and structure and process that exist in Virginia. program evaluation agencies is outstanding It cannot be done by imitation. Nor need it throughout the states. Wherever I go -­ be. Other states have unique cultures, Mississippi in the South, Colorado in the peculiar strengths, and their own ways of West, Kansas, Minnesota and Wisconsin in doing things. the Midwest, and Connecticut in the East -­ Virginia demonstrates not the I am impressed by the quality of the particular way legislative oversight should professional staff. be done (for there is not one way). Virginia In this admirable field, the staff of demonstrates that it can be done. It can be JLARC is second to none. A former student done, and it is being done. And it is by no at the Eagleton Institute of Politics at means rare that it is well done -- not only Rutgers University, where I teach, serves on here in Virginia but in other states as well. the staff of JLARC. I would not have Virginia is a model, but there are also other permitted her to work there, if I didn't models. believe the operation was a good one. Legislative oversight will have its ups Indeed, I'd rather she were unemployed than and its downs, but it will not go away. It working for a shop that did not rank at the will keep on making a difference to the very top. We simply could not afford that effectiveness of programs and performance type of embarrassment at Eagleton! in the states. Right now it is alive and One reason JLARC's staff is so good is kicking, as Virginia so wonderfully that the leadership given it by the Director, demonstrates. Ray ::'ethtel, and the Deputy Director, Phil Leone, has been remarkably skillful. These talented administrators know how to operate as evaluation experts in a highly

Dr. Alan ROBBnthal 4 Status of Sunset in the States

In reviewing the status of sunset in the states and the effectiveness of Virginia's Program Review and Evaluation Act, I will cover three major topics:

• a review of the states that have adopted sunset and their experiences with it,

Rich Jones • a comparison of the Program Review and Evaluation Act with the sunset Program Director processes used by other states, and Legislative Management • observations about the effectiveness of sunset uationwide and the success National Conference of Vn-ginia's Program Review and of State Legislatures Evaluation Act. Current Status of Sunset In the States When considering the current status of sunset, it is important to place it in its historical context. We need to briefly review the political climate existing when it was adopted and the ills it was intended to cure. Such an historical vantage point provides us with a better understanding of sunset's evolution. When sunset was rll'St proposed in the mid 1970s, the uation was recovering from the Watergate scandal. There was a great deal of cynicism about government, politics, and politicians. Many of the reforms adopted after Watergate involved making government more accessible and accouotable to the average citizen. This was the age of disclosure. Candidates had to disclose the amouots and sources of campaign'contributions. Sunshjne laws opened meetings of government officials to public view. And, open records laws allowed the public to see previously secret government rues.

5 St.tus of Sunsst in the St.tss

In addition to the concern about the their laws. The most frequent changes propriety of government operations, this involved the timing of the review cycle, period saw the beginnings of what was to reducing the number of agencies reviewed, become a tax revolt. People objected to the and the types of agencies reviewed. These growth in the size and cost of state changes will be discussed in more detail in a government and supported mechanisms for comparison of Virginia's Program Review reducing government's growth. and Evaluation Act, but essentially these A third factor that shaped sunset's changes were made to reduce the workload adoption was the continuing renaissance of and to make sunset more cost effective. state legislatures that began in the In summary, sunset in some form was mid-1960s. By the mid-1970s, legislatures adopted by 36 states, with at least nine had added staff, reformed their procedures, states having either repealed or and exerted greater control over state discontinued their sunset processes. The budgets. During this period, the first remaining states that have continued sunset legislative program evaluation units were have changed their processes in a variety of formed. ways. In many cases, they moved their These factors combined to make sunset processes closer to the approach sunset an extremely popular concept. currently used by Virginia. Sunset was viewed as another mechanism to open state government to public scrutiny. It provided the means to eliminate state Virginia's Approach to Sunset, agencies if they did not measure up, thus or "I Told You So" saving tax dollars and reducing the size of In 1977 when sunset was sweeping the state government. Sunset was also seen as nation, the Virginia General Assembly the latest in a series of reforms that would created a task force to study the various further strengthen the legislatures' policy­ sunset proposals. The Advisory Task Force making role. Sunset was also a very simple on Sunset of the Joint Legislative Audit and concept to explain, making it easy to Review Commission worked for over one understand by the general public. year and issued a report on the various Sunset's popularity is clearly evident policy options available to the General from the number of states that enacted Assembly. The Task Foree went against sunset provisions and the speed with which both the conventional and political wisdom it swept the nation. In 1976,22 states of the time and recommended that the adopted sunset. The pace slowed after legislature not adopt a sunset proposal 1977, with 11 states embracing it between similar to those enacted by their colleagues 1978 and 1981. In addition to the states in other states. The test of time has which adopted comprehensive sunset review demonstrated the wisdom of the Task programs, several states added termination Foree's judgement and a number of states clauses to specific legislation. are adopting provisions similar to those Sunset has traveled a path marked by contained in the Virginia Act. The Task continuous adaptation and change since its Force's final report listed several concerns inception in 1976. Several states that about sunset and offered recommendations enacted sunset have abandoned the process. to address them. I will touch on only three North Carolina repealed sunset in 1981, and of them tonight. Arkansas, Connecticut, illinois, Mississippi, Montana, and Nebraska have discontinued Termination. Sunset is distinguished sunset. Pilot sunset projects in Nevada and from other types of oversight by the South Dakota were not reauthorized after automatic termination provision. If the 1979. legislature fails to enact legislation In addition, since 1980, all but five of reauthorizing an agency, the agency the states that had sunset have changed automatically terminates. This provision,

Rich Jonss 6 Status of Sun..t in tIIa Statas while making the sunset concept easy to with it. Since 1980, 13 states have made grasp and sell politically, also raised changes in their termination schedules to nnrealistic expectations. The Task Force allow for more time to conduct the stated in its final report, "It will build false performance reviews and to consider the hopes of curtailed government activity and effectiveness of agency operations. This substantial savings where more modest has involved lengthening the review cycles, accomplishments are likely. The purpose of delaying the start of a second review cycle, sunset should be viewed as a periodic and allowing legislative discretion in review, not automatic termination." The determining which programs are to be states' experience with sunset is just as the reviewed and in setting the schedule for Virginia Task Force stated. reviews. One of the recommendations that In a national survey conducted for the NCSL made to the Indiana Legislature, Indiana Legislature in 1983, the National which it enacted, was to lengthen the Conference of State Legislatures review cycle to allow for a more determined that only about 16 percent of manageable workload. Many of these those agencies subjected to sunset reviews changes to provide a more manageable between 1980 and 1983 were terminated. workload are similar to the provisions of Although I do not have a list of all of the Virginia's Program Review and Evaluation agencies terminated, I would be willing to Act. wager a significant sum that the vast Numerous states have also shifted the majority were small, obscure, and relatively emphasis of their sunset reviews away from insignificant agencies. an exclusive focus on regulatory agencies to The Indiana study also pointed out the include other programs and agencies. utah effect that the termination provision has currently subjects all agencies to sunset had on expectations. When asked about the review. West Virginia emphasizes larger effectiveness of the sunset process in agencies, and Kansas dropped all the Indiana, legislators who were unfamiliar regulatory agencies from its sunset cycle to with the process said that it was ineffective concentrate on major program areas. The because of the lack of terminations. Those inclusion of larger agencies and an emphasis legislators who were more familiar with the on cost effectiveness is also a key sunset process stated it was effective and component of Virginia's Program Review pointed to cost savings and other and Evaluation Act which other states are improvements resulting from sunset. It is now embracing. evident from our work in Indiana that those legislators who viewed sunset as a means for Legislative Involvement in the Sunset terminating large numbers of state agencies Process. A third observation of the Sunset were disappointed with its results. Task Force was the need to involve the entire legislature for the sunset process to Workload and Flexibility in Scheduling be successful. The states use a variety of Reviews. Again, the Virginia Task Force approaches for integrating the sunset was right on the mark when it identified the process with the legislative process. Some impact on legislators' and staff workloads as assign the sunset responsibility to standing being critical to sunset's success. The Task committees and do not use a special sunset Force stated, "Any sunset or oversight committee. Others use a special committee proposal must provide for realistic coverage to conduct sunset. Some states, such as and appropriate scheduling so that the Pennsylvania and New Hampshire, use a citizen legislature can accomodate the combination of standing and special sunset workload within its present time frame." committees. Most states that adopted sunset did so Developing a better integration without adequate consideration of the time between the sunset process and the broader constraints and staffing needs associated legislative process is something that many

Rich Jonas 7 Ststus of Sunsst in the Ststss states are trying to achieve. The work of Secondly, as larger agencies are added the Joint Legislative Audit and Review to the sunset schedules, the number of Commission and its central role in the terminations will decline and the emphasis legislative and budget processes helps will shift toward periodic reviews and accomplish this integration in Virginia. improvements in agency operations. In The Program Review and Evaluation many states, this shift has already occurred. Act has been successful in overcoming many Overall, sunset has been a positive of the problems encountered by other innovation. It has not done everything its legislatures when they implemented sunset. proponents said it would but, for the reasons It has provided a comprehensive review of identified by the Sunset Task Force, I don't state government on a schedule that meets think it could have. Sunset has, however, the needs of the Virginia General Assembly. helped to develop the capacity for It avoids the pitfalls and the unrealistic evaluation and oversight in a number of expectations associated with the automatic states where this did not exist. In addition, termination provisions of sunset statutes sunset has increased legislators' awareness and instead focuses on periodic review and and involvement in the evaluation and improvement in agency programs. As oversight process. Even if sunset is evidence of its success, other states with abandoned, the legislators' appetite and sunset are adopting mechanisms that are demand for the type of information provided similar to the Virginia approach. by program evaluation will continue. Finally, as a result of sunset, numerous The Future of Sunset changes and improvements in agency In conclusion, I would like to briefly programs have been enacted, resulting in discuss the future of sunset in the states and more efficient operations and cost savings. to review its effectiveness. Those states that have calculated the costs It is unlikely that additional states will and savings associated with sunset have adopt sunset. In fact, as more states found it to be a cost effective process. complete their review cycles, it is likely In my view, the wisdom of the Virginia that more will abandon the concept. The General Assembly in establishing the key test for sunset's viability will come Program Review and Evaluation Act has when the majority of states complete their been and will continue to be demonstrated, sunset review cycles. I see two things as additional states change their sunset happening. processes to more closely mirror the First, even if the sunset process with Virginia approach. Not only has this proved its automatic termination and scheduled a successful process in Virginia, but it is reviews is abandoned, most states will keep also providing a beacon for other states to the evaluation function. Arizona, Indiana, follow. and Tennessee are examples of states that use sunset as a means of conducting program evaluation. In states such as these, the evaluation function will continue.

Rich Jonas 8 Need for Legislative Oversight & ______L=egislator EX/Jectations of Evaluation Staff

We have all seen "before and after" commercials; well, I am here representing the "before" part. The Oregon legislative system does not yet incorporate any type of program evaluation. I wonld like to take a few minutes to tell you exactly what we are trying to evolve into, and what our limi­ Wayne Fawbush tations are. Let me just mention a couple of the important questions: why we felt this type of operation is needed in Oregon, Representative exactly how we're organized to do this, and Oregon State Legislature what we expect to get out of it.

The Need for Program Evaluation Why is it needed in Oregon? There are basically three reasons: First, we have suffered from a lack of independent investigation capability on the part of the legislature. We found that nobody is going to look after ns, as far as gathering information and trying to make intelligent decisions. The classic example for our state was in the early 1980s, when we were going through severe budget shortfalls and we found a pot of money. That pot of money was held by our state indnstrial accident fund, which we had set up several years before as a semi­ autonomons public corporation. They had about 120 million dollars in a slush fund, and we took 80 million of that. That helped enormously, because we didn't have to raise taxes. You can imagine the furor that action caused. We found ourselves trying to justify why that was still a legislatively controlled body to the general poblic, the Governor, the Secretary of State, and the Commissioner of Insurance. I found myself an interested participant, because I was chair of the committee that had to justify what we were doing. I found myself at a real disadvantage

9 Need for Legis/stive Oversight & Lsgislator Expectstions of Evslustion Stsff in trying to work through the process and We have a research staff of 12 justify our actions, because frankly we had professionals who are basically generalists. no independent capability to either research The office was expanded in the late 19708 our legislative history for intent, or to come when they were given the responsibility for up with viable alternatives that were backed Sunset review, which is sputtering to a stop with facts. in our state as well as others. And we also That was the seed that began to have intermittent committee staff, where germinate in the back of my mind. As I we build up high levels of people during watched the legislative process for ten sessions and then run those down to about years, it became very clear we were going eight people in the interim. We have just to have to develop a more professional staff moved into an objective hiring process and capable of independent evaluation, if we have essentially gotten rid of all of our were to ever stand on an equal footing with political "dead wood" that used to be held the other branches of state government. over in those patronage spots. So we have the core of a good Establishing Legislative Intent professional staff, but they are occupied The second reason program evaluation basically only one out of two years during a is needed, besides independent capability, is two-year bieunium. There's a preparation to try and establish clearer legislative time before the session, but not a lot. They intent. This is perhaps a little more are never sure what we are going to do, and esoteric on the part of the legislative neither are we, so it's hard to prepare. process, but if you have ever tried to go After the session, of course, it takes a while back in history and determine legislative to put out the new laws and grind down. intent, and were successful at that, it was If we are going to continue to be able probably a minor issue that didn't have to justify that type of expenditure for staff, much consequence. A clear statement of I think we need to help make them more legislative intent in the beginning is one of productive. Those of you who have had the the more difficult practices to establish. As misfortune of working in a state where you politicians, we like to have a maximum have large amounts of slack time -- if you amount of flexibility in our decisions. The are interested in your profession and your result is too often a muddled, unmeasurable own peace of mind _4, would probably agree intent. One of the things that we see good with me that one of the worst things that program evaluation doing is pointing out you can run into is a six- or eight-month when we have failed to clearly establish "dead" period when you have to look busy legislative intent. Hopefully such a but aren't. We would like to help you constant reminder will encourage us to be through that situation, at least if you decide more specific. to come to Oregon to work. Those are the three reasons as to why Making Better Use of Staff we are attempting to establish program The third reason is to keep our staff evaluation. fully productive. We meet once every two years, starting January of the odd years, and Dealing With a Structure in Place on the average run about six months. We I think every state is probably a little have built up a very good staff in the unique in the way they organize, basically legislature. We have a fiscal office that because of what they have inherited in the works for the Joint Ways and Means past. Virginia in many ways was an ideal Committee. We have a revenue staff of ten situation. When you began your individuals. They have developed over the reorganization in the early 19708, you were last 13 years very sophisticated computer taking a legislature from the early 1900s modeling, and can provide a detailed into the late 1900s in one quantum jump as printout of any suggested tax alternative. far as building a professional staff

Wsyns Fswbush 10 Need for Legislative Oversight & LegislatOt' Expectations of Evaluation Staff

and upgrading your ability to make Obviously you have to have the legislative decisions. And you did that without having leadership interested, and secondly you need to overlay or compete with an existing the capability to form the question professional staff already in place that had correctly and then proceed in a logical carved out their niche and were protective manner, which we at that point did not have. of that. Oregon unfortunately is noUn that situation. The route we are choosing we are The Question of Legislative Intent taking for two reasons. First, we have This session we tried to introduce severe monetary limitations, and we cannot some reforms of the legislative process to spend any money on a large new staff at this give us a better intent statement. We tried time. In fact, I don't know if we will ever to begin to set up a better and more clear be able to do so. Second, the existing staff legislative intent with the bills that we is well entrenched, with well-justified passed, in order to set the stage for later positions, and we are going to have to deal evaluation. I am sorry to say that those with the structure in place and hope that we bills all died in committee. We are going to can evolve into something that makes more have to try coming at it from the back sense. door: setting up an evaluation function that Two essentials are necessary: (1) The will illuminate shortcomings in legislative leadership must be actively on board, and (2) intent. the staff has to be convinced that this is in So we are back to the position where their best interest in order to proceed. both House and Senate leadership have Fortunately, we have been able to reach a designated a special task force of leadership successful conclusion on both those points. individuals. One important matter decided The time line that we're following will was that we had to have professional give you an idea of what we've been capability to form questions and work through. In 1983 some of us came up with through the methodology. So, we are now an idea that we needed to look at how the looking for a methodologist to staff this legislature was running, and we set up a task task force and coordinate, with the existing force on legislative operations and staff in the building, the formulation and oversight. We did that on the House side execution of these issues. because at that time the Senate leadership quite frankly was not interested in looking at any staff improvements. Identifying Appropriate Issues We began by giving the staff a test The second thing we are going to do is question. The staff was asked to work develop a list of questions. We have found together to develop questions, and to try a that to be one of the most difficult things to rudimentary program evaluation to see if it do in this whole process. If you don't quite could be done with existing staff. We found know how to get where you are going, or very good cooperation, and the staff found what you want, it is very hard to ask an that they were actually interested in intelligent question. As a task force, we stepping outside of the narrow confines they will work to develop several questions that had worked in before. have four criteria: The report was short on a couple of First, it has to be something we can points. It was long on history and short on do. Second, it's got to be nonpartisan. alternatives, quite frankly because we just Third, it has to have a relatively short time did not have the methodology that would frame to be executed. And fourth, it has to enable us to work logically through to the have a positive payoff. Now once we find a conclusions that you are used to in Virginia. question that fits those criteria, we are Going through that rudimentary process going to be in business! Actually, if we get pointed out the need for a couple of things. two out of the four, I'll be happy.

Wayne Fawbush II NfHId far Legislative Oversight & Lagia/ator EJqJ6C18tiona of Evaluation Staff

That's the process that we are Frankly, those types of questions we involved in -- to utilize an already existing, ask, legislators don't have the capability to high-level professional staff for a part-time analyze adequately. Sound alternatives legislature; for us to ask semi-intelligent would also lead to less risk-taking through questions; and for the staff to give us hack legislative action, and hopefully more intelligent alternatives. evaluation of those issues. Rather than saying, "Awl shucks let's try it and see what happens; after all, we only have to wait two Expectations years to see if it works and then we can What we expect is essentially to make change it," we could say, "Well, this may be this committee on oversight a permanent a good idea; let's evaluate this and come committee composed of leadership back next session with concrete proposal." individuals who will have some tenure. If Also, I think we need the independence we've learned anything from Virginia, it is from the other two branches of our that you have to have secure, tenured government. We need the independence to individuals on that committee for continuity. ask questions in our way and get the answers The other thing that we're interested' from our staff for our use, and not be in is having good policy alternatives lined dependent upon either the Governor or up. For example, we have probably passed Secretary of State and their audit functions more anti-drunk driving legislation than any to provide that information for us. three states combined. We've done just Lastly, and I think most importantly, I about everything I can think of to think it will make our staff in the building discourage drunk driving, and we're not much more productive, for a couple of quite sure what works and what doesn't reasons. First of all, rather than simply work. We would like to know, or at least reacting to problems, hopefully we will be have some idea of, the consequences of the able to take the initiative. People work multitude of actions we've taken in the last better if they're taking the initiative and seven years. working through something rather than I think the same goes for the catch simply having to react. And we can say, phrase of "economic development." I don't "Yes, we do use our staff 24 months out of know how many of you are caught up in the the biennium," and "Yes, our staff are much rush to develop economically and to prove happier because they are now productive," that the legislative body can lead the state and "By gosh, we actually use what they in economic development. We need to step produce." That is hopefully what we will back and ask a question: are we leading, or achieve at the end of this process. are we getting in the way of a natural process that maybe would be better off without us? @xmW%-----

Wayrl8 Fawbuah 12 Need for Legislative Oversight & Legislator Expectations of Evaluation Staff

I would like to discuss what we as legislators ought to want from the auditing and staff evaluation function -- an answer to the "what is it that you people really want?"question. Sometimes legislators need to take a step back, because legislators themselves need to think through what it is they really do want from a legislative auditing or evaluation function. Sometimes what, at initial blush, they might think they Mordecai Lee want is not necessarily in their own best interest in terms of their long-term role as legislators and the long-term role of a legislative evaluation staff. Senator Perhaps what we are dealing with can Wisconsin State Legislature best be described as a clash between different cultures. Think of politicians as members of a tribe that is incomprehensible to outsiders. We as politicians need a Margaret Mead to come in, wear her grass skirt, and try to understand what our culture is. Then she could go back to the world on the outside and try to explain what we need, how we operate, how we make our decisions, and what is the best way to serve us. I suggest that the best way for a legislative eval­ uation staff and structure to function within the legislature is not to be integrated into the legislature but rather to be separate from it. And, that the way to help the legislature is not be coopted by it, but rather to be autonomous from the legislature. Now that is a very difficult, tricky, and tough way to accomplish a good relationship. But I suggest that, in the long run, that is the way to best serve legislators, even though we might not think so. Defining the Relationship A good example of the relationship I'm describing can be found in a biblical phrase. The Bible describes how Eve was created

13 Need for Legis/etive Oversight & Legislstar Expectetions of Evsluetion StBff

from a rib of Adam. The way the original agenda or conflicts between the legislative Hebrew was translated was that God branch and the executive branch. created Eve to be a "help meet." I think You want that kind of nonpartisan nobody was ever quite sure what that objectivity, but at the same time you want means. Well, since I am Jewish, my parents timeliness. What is it that most helps the forced me to go to Hebrew school for what legislature: timely information, timely seemed to be an interminable number of reports, ones that fit into the legislative years. Luckily, I'm able to understand that cycle. We don't want a study of a subject to phrase in the original Hebrew, and I don't come out a month after the bill is finally think it's translated very clearly. signed by the governor. We want that study Interestingly, in the original Hebrew, the to be issued one month before the role that God gave Eve when He created her legislative process begins working on that out of Adam's rib can be translated as "a issue. helper against him." So at the same time that we want the Now what does that mean? I suggest legislative audit function to be independent, that it might indeed summarize what a good objective, and nonpartisan, we also want it marital relationship is. It's a relationship of to be sensitive to the realities of the helping somebody, of being intimately tied legislature, and to try to time its work to fit with them or connected to them, inexorably into the timing of the legislature. tied. But at the same time the two are separate and apart from each other, having Responsiveness vs. Initiative a sometimes contentious and conflictual The second pair of conflicting relationship. There are time~ when the way principles that I suggest we need in a good to help one's spouse is by being apart from relationship between the legislature and its them and at times even against them. I evaluation staff is that, on the one hand, we suggest that this is the way a good as legislators want that staff to be legislative evaluation staff can help the responsive to us. We want them to do what legislature, even if the legislature isn't we are interested in. But at the same time, quite sure that's really what it wants. in total contradiction, we don't want to be Perhaps what I am describing is handcuffing them and preventing them from impossible to accomplish. Nor am I doing other things -- things which suggesting that we in Wisconsin have politically we may not want them to do, but accomplished paradise on earth. Rather, we in terms of the policy process would be in Wisconsin have accepted as a premise relevant. In other words, even though we what the relationship ought to be, and we want them to be responsive to us, we also try constantly to work towards that goal. want them to be able to initiate, on their own accord, that which would be helpful Objectivity vs. Timeliness even if it is politically sticky, touchy, and I suggest that there are seven sensitive. We want them to have the conflicting principles, or pairs of principles, autonomy to initiate those studies which are that a legislature and its evaluation staff in the public interest. ought to hope to accomplish in their relationship. The first pair of principles Disinterest vs. Advocacy that ought to exist between a legislature The third pair of principles is that on and its evaluation staff has to do with the one hand we as legislators want to be objectivity. One wants a nonpartisan and sure that the information that we get from objective evaluation function to occur -­ the legislative audit staff is definitely one which is separate and apart from the disinterested. On my desk in the state partisanship of the legislature, separate and capitol I am constantly getting reports that apart from the agenda of the individual are interesting to read, but the first thing I legislators, and separate and apart from the do is to check where they're from. And if I

MtXdecei Lee 14 Need for Legislative Oversight & Legislator Expactations of Evaluation Staff will quickly ascertain what group mailed it legislative evaluation staff to be relevant to to me, I often don't have to read the report, what is going on at that time in the because I can guess the ending. There is so legislature. I want a sense of immediacy much self-interested information in the about what they are doing. After all we, as political process, so many people with axes legislators, don't benefit from their doing to grind, so much selective information, so something which is not relevant to what we much manipulation and number-crunching to are worried about that day, or week, or accomplish a preconceived conclusion. All month, or biennial session. of that is predictable and not very useful in Yet on the other hand, we don't want the legislative process. How few sources of a legislative evaluation staff to be so information we as legislators have that are focused on immediacy that what we are truly disinterested, that do not have an ax getting from them is short-term to grind. That is why in Wisconsin it is a information, as opposed to information that pleasure to get an Audit Bureau report, embodies long-term trends and gives because I know that it is disinterested long-term policy advice to the legislature. information. So, once again the internal contradiction: Yet even though I want to be sure that we want the immediacy, yet at the same the Wisconsin Legislative Audit Bureau, or time we want the long-term trend any other legislative evaluation staff, is a information. disinterested observer of the legislative process, at the same time I don't want them Quantifiable vs. Nonquantifiable to be merely passive observers. We don't Another pair of contradictions: On want a good legislative evaluation process the one hand, we as legislators want to to be so disinterested that the staff is maximize objective information by getting a sitting in a glass booth, and is unconnected lot of quantifiable data from the legislative to the process itself. Even though we don't audit bureau. After all, "numbers don't lie," want the staff to have an ax to grind, even right? But at the same time we want that though we want them to be disinterested, at legislative evaluation staff and our the same time we want them to be colleagues to understand that numbers advocates for their own disinterested aren't everything, that what you can information. quantify isn't necessarily the whole picture. Many times an audit or evaluation will So at the same time that we want objective, be submitted to the state legislature, and a quantifiable information from the special interest group or an individual legislative evaluation staff, we also want legislator with a self-interest will the opposite. We want them to try to selectively package the results of that evaluate that which is not quantifiable. We audit, citing it to prove a point that the don't want them to become enamored with audit actually doesn't prove. As much as I numbers to the point that they lose focus on don't want the legislative audit bureau to be the big picture, or that they're only looking involved in the political process, I do want at half of the problem. So we want both them to be involved in the legislative quantifiable and nonquantifiable process to prevent the manipulation of their evaluations. information. I want them to be advocates for their own disinterested position, which Geocentricity vs. Nongeocentricity again is an internal contradiction and The sixth pair of conflicting principles difficult to accomplish. concerns geography. On the one hand we want to avoid any kind of geocentricity. We Relevant vs. Long-Range don't want an exclusive study of how things The fourth set of principles that are going in our state, under the embody an internal contradiction is that, on presumption that our state is always doing the one hand as a legislator, I want a things right, or that we can't learn from

Mordac8i Lee 15 Need for Legis/atiVl/ Oversight & Legislator Expectations of Evaluation Staff other states. Yet, on the other hand, we Reidentifying the "Client" don't want to get on a treadmill with other In Wisconsin, in the environmental states -- doing something just because other protection function, we have a very states are doing it. Just because we are interesting analogy for what I am 49th in the poll on the XYZ problem doesn't prescribing for auditing. In the area of mean that we ought to be competing with environmental protection, we have Mississippi so we can be number one. So we something known as the "public intervenor." want both the benefit of nongeocentricity as The public intervenor is an assistant well the benefit of geocentricity. Go try to attorney general who is answerable to no solve that paradox! one, is appointed by an advisory committee of enviroumentalists, and whose only "client" is that one participant in the policy Comprehensive vs. Usable process that is not organizable: the The final pair of conflicting principles long-term interests of the environment in a good relationship between an evaluation itself. The job of the public intervenor is to staff and the legislature is that on one hand fight the executive branch, the legislative we want comprehensive information, yet on branch, and the judicial branch for the goal the other hand we want usable information. of environmental protection. There is too much of a possibility that a As you can imagine, she frequently is comprehensive report will overwhelm the in conflict with those more organizable policy and legislative process, and become interests in the policy process, whether they so "cosmic" that it is not usable within the be business interests, local governments, or legislature. Again, we want a-balance even state government itself. These groups between two principles. On one hand we are very often unhappy about some of the want comprehensive information, but on the lawsuits she brings or some of the testimony other hand we want it to be'usable in the she makes. But she is representing the only legislative process. After all, in the interest of the public policy process that is legislature we deal with very specific not organizable, and that's the long-term questions. We vote "yea" or "nay" on very interest of society-at-Iarge. specific bills and very specific issues. I suggest that the legislative What I've suggested is that every state evaluation staff, even though its immediate needs to try to accomplish a delicate boss is the legislature, should think of its balance between an evaluation staff and the other boss as the one participant which is legislature. Perhaps like the relationship not active in the policy process: the public between two spouses, this is a goal one interest. And if, indeed, we are to be able never quite accomplishes, but rather always to develop an understanding that a works towards. After all, the perfect legislature is best served not by having the marriage is not one that stops. The perfect evaluation staff supinely subservient to it, marriage is one that continues working but by having a high quality professional towards the goal of a perfect marriage. evaluation staff which views its client as As tempting as it may be to politicize including the long-term public interest, then or handcuff an evaluation staff, it is in the I think the whole legislative process will long-term best interest of the legislature to benefit in the long run. accomplish the kind of autonomous and independent relationship I've tried to outline. M@I))))IM

Mordecai Lee 16 Need for Legislative Oversight & Legislator Expectations of Evaluation Staff

My purpose ill to share with you 8 few thoughts about Florida and the development of our auditing process and our performance review process. Hopefully I can help you understand how it ill an integral and a very important part of the Florida oversight process, which involves not only this function but a variety of other functions that are a part of the legislative process Herbert F. Morgan itself. An Unusual Beginning First of all, you may f"md interesting and somewhat humorous the actual Representative development of our present Auditor General Florida State Legislature in the State of Florida. Until 1967 the Auditor General was known as the State Auditor. He reported to the Governor and cabinet in the executive branch,and he audited the executive bmnch. Of course, for purists there were the traditional complaints ahout the executive branch auditing itself. But despite that, there was a considerable amount of credibility because of the people who occupied the function. However, in 1966 Florida elected its first modern-day Republican governor, Claude Kirk. This was the first in many years in Florida, a state that ill dominated at the present time by the Democratic Party and has been for a number of years. The legislature, I think, had its attention drawn to the fact that perhaps now there was a need to understand better the importance of the true separation of powers between the executive and the legislative. As a result, they passed an inoocuous little bill in 1967 led by Representative Lou Wolfson of Dade County, and of course with the full support of the Speaker of the House at that time, Ralph Turlington, and the President of the Senate, Verle Pope, Jr., that was known as "a bill creating the office of legislative auditor."

17 Need for Legislative Oversight & Legislator Expectations of Evaluation Staff

Believe it or not, the Governor and categorized by many as one of the most everyone else -- since there was no independent legislatures in the country -­ testimony, no real presentation, no perhaps the most independent -- because of flamboyant speeches on the floor -- thought our ability to call ourselves into session and this meant the legislature itself was going to meet in the interim. We organize to be audited by this person. So the bill immediately after elections in November, became law and went by the governor and we are totally and completely staffed at without having to even override his veto. the substantive committee level and at the Shortly thereafter, however, as soon as the appropriations and finance tax committee act took effect, the State Auditor at that level. And we have the fully staffed and time, Ernest Ellison, resigned his executive very competent Auditor General's office branch post to assume the duties of assisting in our oversight function, as well. legislative auditor. Shortly after that he To understand how that works in the was successful in enticing away most of the oversight function in the State of Florida, it staff, most of the records, and all of the would be important, 1think, to see in information. We then had a legislative capsule a little of how our process works as auditor of Florida who had everything and it relates to our state budgeting and was now a part of the legislative branch. spending, because obviously a good portion That in itself could create a bad of what an audit function does is look at impression in your mind. The humor of it that. But you also have to look in the draws attention to the fact that the existing performance area -- at the actual situation was not perhaps the best way to operations and how the agencies and the have an audit function set up, that is, various entities of government are housed in the executive side. But by the accomplishing the objectives of the task same token, the method of accomplishing that we set for them. the change might lead you to think that the The appropriations process in Florida function is domiuated by the legislative side depends upon the work of the two and is therefore still not independent -- one appropriation committees, one in each of the tests that ought to be a very integral house. We do have a recommended bill from part of a good audit effort. the Governor, but we do not introduce a That is not trne, however, because governor's bill in the legislative process. shortly thereafter the name was changed to Instead, we develop the committee bill the Auditor General of Florida to avoid any through the legislative process itself. The confusion and to make it clear that the governor's document is only an advisory function of the office was to be a totally document and represents his best independent and totally capable entity, suggestions on how we should develop the designed to look at the operations of state state's budget and spending effort. government in Florida. It was to do so in a The legislature, being competently tmly independent manner from the staffed at that appropriations staff level, is executive branch, which is obviously where able to analyze those documents and those the primary effort would be directed. And needs of state agencies for spending it would be directly responsive to the priorities and develop what we believe is a legislature without being domiuated by that trne priority spending document that body, and without the legislature dictating appropriates federal funds and grant funds, the political aspects of the results. That as well as general revenue funds. And we perhaps gives you some idea of the audit all of those various programs, as well. placement within the legislative branch. Oversight in Florida involves a number of entities. First of all, Florida believes Oversight in the Larger Process that any state needs an adequate and strong In Florida we believe we have a very oversight function, and, as a major element strong legislature. We have been in that function, there must be a strong

Herbert F. Morgan 18 NfHld for Legislative Oversight & Lagislator Expsctations of Evaluation Staff audit process with particular emphasis upon division of performance auditing. performance as well as financial and So the emphasis within Florida has compliance auditing. Both houses of the gone from almost nothing to one of the Florida legislature have oversight functions largest performance auditing capabilities in that involve several committees: the the country. And the reason for that is, appropriations committee itself, regulatory very simply, the desire of the Florida committees of several types, governmental legislature to have the kind of objective, operations committees in both houses, and independently arrived-at information, oversight subcommittees of every in-depth and yet in a form the legislature suhstantive committee of the Florida House can understand and utilize in a timely of Representatives. This has heen the norm manner. for the last four to six years. The staffing for the performance audit However, the major actor in this capability, unlike the rest of the audit process is the largest organization within function in Florida, uses the backgrounds of the legislative process for oversight: the a numher of different professions. For Office of the Auditor General. There have instance, economists, statisticians, public heen some 4,OOO-plus financial compliance administrators, accountants, MBAs, audits completed since 1970, and these educators, engineers, and others are utilized audits have found a numher of ways to save to staff that particular function and ensure money, to reduce spending in some that we have the expertise to make the instances, and to deter fraud and abuse reviews that are necessary. where it has heen found. The auditing committee in Florida is the primary policy regulator of that process, An Increasing Role keeping in mind, however, that our Auditor A lack of analytical information has General is independent. The Auditor led to a real interest on the part of the General's office can initiate audits on its legislature in Florida over the last six to own, does not have to wait for direction eight years for an increased effort in what from the Committee, and has an on-going we call our performance auditing function. level of operation at all times. But the role That increased effort has led to an of the Committee is to provide policy increased emphasis on the role of the Joint direction through control of the budgetary Legislative Auditing Committee. process for the Auditor General. The The committee is composed of five budget of that audit activity is approved memhers from the Senate and five from the through this committee as an annual House, and the chairmanship rotates activity each year. The Committee's role annually. The House has the chair the first also mandates adoption of the personnel job year of the biennium, the senate the second classification and the pay plan for the year, and we are about to make that Office of the Auditor General and, of transition shortly. The five memhers from course, direction in audit topics that the each house, however, serve for the full legislature is interested in. two-year term. Those appointments are made by the Speaker and by the President. Key Processes The Joint Audit Committee is staffed We have formalized the process for by a staff director and a committee requesting audits by other parties, including secretary and calls upon the talents of the other legislators, so that Office of the Auditor General's office in coordinating the Auditor General is not overwhelmed with work of this committee as we look at the requests for audits or performance reviews needs for audits. The office of the Auditor of one kind or another. Otherwise, it would General of Florida currently has almost 600 he impossible to meet the test of positions authorized; 344 of these are in the timeliness. Topics not really appropriate auditing division, and 68 are located in the for audits but needing attention can he

Herbert F. Morgan 19 Need for Legislsti"" Oversight & Legislstex Expectstions of Evslustion Stsff

directed by the Joint Auditing Committee both honses of the legislature, and his to other areas, perhaps to appropriate removal would require the action of both legislative staff or even to areas within the honses. Therefore, he is insulated from the executive branch where they might be political interference of any individual appropriate. legislator, and even of this committee. The primary role of follow-up Yet he is not autonomons to the point coordination of audits and performance that he feels he is another branch of reviews is the responsibility of the Joint government. He mnst keep in mind that his Auditing Committee. And the primary duty work product is not a game that he is of this committee through our staff director playing but an integral part of the overall is to ensure the processes work to do that. process, designed to make Florida We might take up an investigation of any government function better, and to make matter within the scope of an audit, either the legislative process able to really set the completed or being conducted. The priorities for spending where there are committee can refer an issue to either the limited revenues and unlimited needs. Speaker of the Honse or the President of the We expect excellent, sound, Senate. Also, we have referred issues to well-researched management information. substantive committees and are in the We want to know if programs are operating process of formalizing more directly a efficiently and effectively. We want to process for including a "sureness of review" know if the underlying assumptions for of what we call "adverse findings" in audits programs are appropriate. And we want to by our substantive committee processes and know if program staffing is consistent with by the legislature itself. This process is goals and objectives. particularly objective over historical time Let me give you jnst two or three periods, where we can see if there are examples of where we found these repetitive patterns -- issues that arise first particular items to be especially nseful to at one agency, then at Mother. ns. In many instances they have been This adverse findings tracking system, equally controversial, even in their as we call it, is being computerized at the utilization. In fact, the work of this audit present time. We will have the ability staff has been attacked by outside forces, shortly to access this kind of information in not jnst agency people, but others that felt a very timely manner and see if we have they had something to lose from the process. properly followed up. When that is In early 1980 there was a performance developed and coordinated with a portion of audit of the administrative structure of the our budget work through the appropriations community mental health service delivery committee, the appropriations staff itself system. lf you don't believe that can get will be able to easily access the hot politically, you need to know something information. They will be better able to about community mental health in Florida. scrutinize and analyze each agency's budget And I am sure most states are the same, and request and the governor's recommendations it does get hot politically! to understand what it is we need to fund, The audit found that there were two and where our priorities should be. distinct administrative structures. They had identical functions, and this led -- finally, Legislative Expectations after gnashing of teeth -- to the enactment What does a legislature expect of the of legislation eliminating the duplication performance review function or audit and budget savings of some $700,ooo-plns function? In Florida we are committed, per year. And, since government tends to number one, to an independent auditing grow from year to year incrementally, that function. Although the Auditor General is $700,000 would have grown in the future. reviewed through this committee, the That's one example in a particular area that Auditor General's appointment is through I think is important, and the document

Herbflrt F. Morgan 20 Need for Legis/etive Oversight & Legis/etar Expectetions of Eve/uetion Stefl behind that was the report of the Auditor But the performance review by the Auditor General. General was the basis on which the House of One that drew equal controversy was a Representatives provided the leadership to performance audit of the state insurance essentially deregulate trucking in the State regulatory program. In this instance we of Florida as it relates to in-state trucking looked at an operation under one of our activities. And now we find that none of elected officials, and therefore drew the ire the "horribles" that were forecast have of that official because it criticized the developed, and in fact the consumer is very operation within that agency. One of the adequately served in Florida. most significant findings was that most of But you can understand the the Department of Insurance's consumer controversy associated with that and the services resources were allocated to difficulties in even getting the legislature operating 21 field offices to handle itself, without the independence of that consumer complaints. You need to Auditor General, to focus its attention on understand that those field offices each that. The legislature was willing to say "go were staffed with several people, and they and look at it." When the information came were located strategically around the state back it was unbiased, it was available, and it for an elected cabinet official. I am sure was a matter of record to be utilized in a you are beginning to develop a picture, and good process. you can see how important that consumer complaint function was through those 21 The Linkage With Appropriations complaint field offices from a political I want to mention the tie between the viewpoint. appropriations process that I alluded to The audit noted, however, that the earlier and the Joint Auditing Committee. Department of Insurance First of all, the appropriations process is complaint-handling activities could be essentially future-oriented; that is, the first centralized in order to reduce cost without order of business is to complete the next affecting service. It noted that 21 offices year's budget. We do annual budgets, and I were not totally necessary, and that the was Chairman of that committee for six toll-free phone lines, some advanced years. Also the budget process is limited consumer education activities on a more usually to a one- or two-year time span. limited scale, and some other activities It's often more incremental than we like to could accomplish the same results. admit, and it uses the existing government As the direct result of the audit, 38 structures as a base. positions and almost a million dollars were So the appropriations process, as good eliminated from the Department of as it is -- from a former appropriations Insurance's budget over a two-year period. chairman who loves it and has a great deal The Bureau of Field Operations was of pride in it -- cannot provide that kind of reorganized from 21 field offices to six, and objective, stand-off review that an has implemented what we believe is a better audit-type performance review function can load of services. provide. I don't see how the state could The last example that I would like to operate without it. I don't see how the mention is perhaps the most controversial, State of Virginia could operate without what and may have led the way in utilization by I know to be one of the finest processes of the legislature. It relates to trucking this kind anywhere in the states. Virginia is deregulation in Florida. An audit and a a state that has a reputation nationally, and performance review were done. It was very certainly we have learned a lot there, as we controversial; it was attacked by the all learn from each other. trucking industry itself; it was attacked by But the Auditor General's many legislators; it was attacked by major performance review process complements business interests, and lots of other people. that appropriation process, because it grants

Herbert F. Margen 21 Need for Legislstive Ovsrsight & Legis/etor Expectations of Evaluation Staff what we call "formal status" to program the support of the Speaker and the President evaluation as a function. It also insulates and the leadership of both houses, I think we the program evaluation from both executive are going to find that our function is that and legis1ative politics. Budget committee much better as well. staff can expect to receive what we call nonpolitical audit reports -- objective, The Legislature's Rightful Role timely, carefully arrived at and documented. Is it important for the legislative We are not obligated -- and I don't process and strength? I think that it is want you to think that I believe as a absolutely essential. I am a great believer legis1ator I am obligated -- to accept and a strong proponent of the legislature everything the Auditor General tells us. I exercising its rightful coustitutional place in don't. Sometimes I think he's wrong. the process. The erosion of the powers of Sometimes I think he may be right but it is any branch of government is a serious politically impossible to do what is coustitutional question at the national level recommended at the time. Those are the and at the state level. And the failure of practical realities of the political world that the legislative branch to properly exercise we live in. And then sometimes he is right the powers that it has coustitutionally is in and time is not on his side. It may take a my estimation a usurping of our duties as few years to accomplish the objective, or elected officials, because we are the only the system may have to run it through a people in Florida -- and I think this is true cycle. in most states -- that are charged with But the point is that we can have that taxing and spending policy decisious. No kind of document and we can have specially other branch of government is given the trained staff who can focus their full-time latitude to determine those particular policy efforts on that kind of evaluation, rather initiatives. than being pulled off, perhaps, to develop To me a strong performance review budgets during the 60-day period. It and an audit function are a major, integral strengtheus the budget process. And as I part of our process, so that we can be that mentioned, when we fully coordinate and true co-equal branch of government that we bring together the adverse findings tracking should be. It protects that co-equal status, system, and a few other activities of the and protects our ability to make the kinds of budget process, I think we are going to find policy decisious that must be made only by that the system will work even more the legislature. effectively. As our joint auditing committee matures even more in the new §~ role that we have cut out for ourselves with M11m DDt

Hsrbart F. Morgan 22 Oversight in the Congress and the U.S. General Accounting Office

When Ray Pethtel first asked me to talk about issues and trends in evaluation these days in Washington, I was reminded of that old New Yorker cartoon some of you may remember that shows a sign saying: Entering Hmsdale, Founded 1802, Altitude 620, Population 3,700, Total 6,122. In other Eleanor Chelimsky woI'ds, I'm not sure the evaluatol'S' data are always used as we'd like them to be. Director, Program Evaluation On the other hand, I do want to report and Methodology Division progress -- indeed, considernble encom-aging progress -- in many areas of U.S. General Accounting Office evaluation. So, to tell you about that, I thought I'd concentl'8te on two things Past President today. Fil'St, I want to draw a general Evaluation Research Society picture of progrem evaluation as I see it being prncticed today in this country, and, in particular, of legislative progrnm evaluation at the federal level (that is, evaluations perlOI'Illed to seNe policymakel'S in the Congress). Second, I want to tell you of the ways in which I've seen evaluation used by the Congress, and the effects of that use of the executive branch of government. To do this, I'd like to tinger a little bit over the origins and development of progrnm evaluation to give you a bit of context (after all, it's hard to know where we are if we don't know where we've been) -- to say what I mean by the teI'Ill "progrnm evaluation", and then talk about its pw:poses, uses, and importance. In the process, I'll be commenting on the GAO role in meeting congressional evaluation needs, and summarizing the findings of three GAO evaluations so as to trnce their effects on legislative and executive branch policymaking. Origins and Development of Program Evaluation As everyone knows, I think, program evaluation is neither very new nor is it very revolutionary. In fact, it's been around for

23 Oversight in the Congress end the U.S. Generel Accounting Office

a long time, developing slowly under many The second path was different in that different guises and disguises. Some people it involved fields like education, public trace its origins to the Age of Reform in health, or crime and delinquency (rather England and the year 1870. In that year, than defense), and it antedated the first evaluations of educational achievement path since, already by the 1950s, large-scale brought some pretty contemporary retrospective evaluations were being recommendations for "incentives" to commouly performed, using methods of teachers in the form of -- guess what? applied social research such as the survey, pay for performance, or "payment according or computer-assisted statistical analysis. 5 to results." 1 Other people think history's Also, this path didn't belong to anyone first evaluation took place a little bit discipline; iostead it received major earlier, in the days of Nebuchadnezzar, methodological contributions from a broad when one of the King's stewards conducted array of fields including psychology, a controlled experiment to test the effects sociology, economics, political science, of a vegetable diet on the "fairness" and applied statistics, and anthropology. "fatness" of Daniel, Shadrak, Meshak, and Little by little over time, of course, Abednego. Z . the two paths have become much less Whatever the actual origins of distinct, and today it's the most ordinary program eValuation, it's clear that interest thing in the world to find a mixture of in the effects of government programs and techniques from both paths used together in policies goes back a long way in the United a single study.· So the yield for modern States. But the ability of evaluators to evaluators has been an increasingly rich respond to that interest is What's new. The repertoire of methods that we can use to state of the art has developed remarkably answer different types of questions about over the past 20 years or so, and I think this government policies and programs. is the result of two quite .disparate and The patterns of evolution have also independent paths of inquiry. been different for the two evaluative paths. One of these paths was the 1950-ish The applied social research path developed effort to rationalize the management and slowly but surely and steadily, driven first resource allocation of defense missions and by the expanding Great Society and its "War programs. We used to call that PPBS, or the on Poverty," which generated a need to find Department of Defense's Planning, out what effects these unprecedented social Programming, and Budgeting System. But programs were having, and later by a ouly one component of PPBS involved the developing concern about the costs and retrospective activities of program effectiveness of these programs, which evaluation; the main thrust of the effort triggered efforts under Presidents Nixon and was really on planning, so that the Carter to gain improved efficiency through techniques PPBS fostered (like policy evaluative information. analysis)· were tailored more to establish PPBS, on the other hand, didn't likely future effects than to identify the develop slowly and steadily. It exploded on actual, observed effects of implemented, the horizon, full-bodied, you might say, existing programs or policies. The question from the head of McNamara. It was barely was "what'll happen if we implement this a glimmer on the national consciousness in program?" rather than "now that the 1960. But by 1965, its success -- or at least program is implemented, what happened as its proclaimed success in DOD -- was such a result?" The PPBS techniques -- policy that President Johnson ordered it analysis, cost-benefit analysis, cost­ implemented in all agencies of government. effectiveness analysis, systems analysis, Now he did this despite the fact (and here I operations research -- were all developed quote Aaron Wildavsky) that "not a single by economists and all had economics as study of this important experiment was their core.4 undertaken before the decision was made to

Eleenor Chelimsky 24 Oversight in the Congress end the U.S. Generel Accounting Office spread it around the laod."7 Now I don't say this to belittle the By 1970, only 5 years later, much of considerable growth that did occur, simply the excitement had faded as people begao to to note a difference in the type of growth recognize the technical prematurity of some from that experienced by PPBS. The fact of their efforts aod the remarkable opti­ is, with regard to the growth of applied mism of some of their expectations. What social research-type evaluation, that over happened was that once again, as with the single decade spanning the late sixties performaoce budgeting before it, the to the late seventies, the federal funds assumption had been made that the needed spent on non-defense program evaluation data systems aod measures with which to rose from about $20 million annually to evaluate program outcomes could be quickly about $180 million. That number moves to aod easily developed. over a billion when you count in c ~fense Now, of course, with the benefits of evaluations. The number of people who used 20-20 hindsight, aod with the knowledge to be able to lay reasonable claim to the brought by 20 additional years of technical title of program evaluator went from a experience, it seems almost staggeringly haodful to several thousaods over the same obvious that the size of the data base aod period. "The number of studies, the number measurement infrastructure that were of evaluation units in government agencies, needed for something as big as the number of private research firms, the "government--wide PPBS" called for academic departments having programs in lengthy research aod development. The fact evaluation, the number of legislatively that there had been little, if aoy, prior maodated requirements for evaluation, aod development aod testing of measures that the actual use of evaluation findings by could validly represent program outputs was legislators aod maoagers all increased ao obstacle to planning aod evaluation that dramatically during this brief period." 8 PPBS just could not overcome. Even more I think it's importaot to note that importaotly, I think, the lack of ao while the problems of PPBS did end in the evaluative information base that could system's decline, they nonetheless helped speak to actual experience with existing the development of program evaluation in programs terribly weakened the credibility that they confirmed aod clarified for maoy of estimates for the future made by PPBS' policymakers aod aoalysts the basic, various aoalytical techniques. In effect, enduring, governmental need for a program measuring future costs aod benefits of evaluation information base. You know, alternative policies aod programs requires people often say that one of the lessons at least some knowledge of current effects. from PPBS was that perfect scientific I think the slower growth of the use of rationality may not be achievable in applied social research helped its government. Well, I'd say that that was no development in comparison to what startling revelation. Some of us, in fact, happened to PPBS. Also, there was a had already suspected that. tendency toward small rather thao gigaotic What was really crucial was the applications (I meao projects or programs, discovery that the ordinary "muddling rather thao systems), aod there were through" of the political process also infinitely less lofty expectations. So this benefits from sound information, especially allowed the applied social research path empirical information about past program more time to develop ao understaoding of performaoce, as a regular part of normal, which initiatives could aod couldn't yet be incremental policymaking. So PPBS undertaken, time to increase the demonstrated both the need for ao development of needed support systems evaluative information base aod the fact (such as data bases), aod time to sharpen its that it didn't exist. At the same time, the focus on the professional needs aod training progress being made in applied social of the developing interdisciplinary field. research showed that such ao information

Eleenor Chelimsky 25 Oversight in ths CongrsBB snd ths U.S. Gensrsl Accounting Office

base could be developed -- slowly and piecemeal, perhaps, but steadily and cumulatively -- over time. Definition, Purposes, The fact that many policymakers had and Importance become aware of this is reflected in the numbers of administrative data bases that Of Program Evaluation were developed in the executive branch over How then do we define program the 1965-1975 period. The most important evaluation today? The definition we use at purpose of these data bases, of course, was GAO is that evaluation is the application of to allow the monitoring and evaluation of systematic research methods to the government programs. They include such assessment of program design, imple­ topical areas as education, employment, mentation, and effectiveness. That is, it health, energy, the environment, public gives information on program design for the assistance, and crime and delinquency, to purpose of policy formulation -- and by that name only those. Then there was Title vn I mean assessing or justifying the need for a of the 1974 Congressional Budget and new program. It can provide information on Impoundment Control Act, which gave great program implementation for the purpose of impetus to legislative program evaluations, policy execution -- and there I mean making as it did a very strong statement ensuring that a program is operated in the about the importance and utility of program most cost effective way. And, finally, evaluation to legislative policymakers. This evaluation develops information on program was the legislation that directed the effectiveness for the purpose of General Accounting Office tl) develop its accountability in public decision-making -­ activities in this area. and there what I mean is determining the So, to make a long story short, I think effectiveness of an operating program and that two parallel paths of i,nquiry have establishing whether it should be continued, contributed powerfully to the development modified, or terminated. So evaluation can of program evaluation as we know it today. address itself to all three purposes. One path proceeded rather gradually, across Now you can see right away that these the multidisciplinary fields and methods of purposes -- policy formulation, policy applied social research. The other had more execution, and accountability -- have of a bell-shaped curve. Both paths definite implications for the kinds of pioneered or used and diffused methods (for questions evaluations may be asked to example, surveys, case studies, controlled answer. For example, let's look at the experiments, statistical research and purpose of policy formulation. It applies analysis, longitudinal design, meta­ essentially to new programs and it may need evaluation) which have become part of the answers from evaluation that include: everyday language of program evaluation. More and more people are now trained in the use of quantitative techniques generally • Information in the problem addressed and social science research methods in by the program. (How big is it? Do particular. As time passes, consensus has we have four anecdotes about the been building in the field in many areas. problem or do we have information This is especially true with regard to the about its frequency and direction? definition and purposes of evaluation, but How is it changing? Do we really need also with the recognition of its usefulness a new program or new legislation to for both executive and legislative branch solve this problem? If we do, how policymaking and management needs. likely is it to succeed?)

Elsen", Chs/imsky 26 Oversight in the Congress end the U.S. Gen.,el Accounting Offias

• Information on the results of past • Information on program outcomes. programs that attempted to deal with (What happened as a result of the problem. (Were those programs designing the program and feasible? Did they work? What implementing it?) difficulties did they encounter?) • Information on the degree to which • Information that tells us how to select the program made, or is making, a one alternative program over another. difference. (That is, what favorable (What are the likely comparative costs change in the problem occurred that and benefits of going one way versus can be directly attributed to the another? What kinds of growth program?) records were experienced by different alternatives in the past?) • Information on the unexpected (as well as the expected) effects of the program. (E.g., was a program of When you move to the purpose of drug education tocombat drug abuse policy execution, you get entirely different accompanied by an increase in the use kinds of evaluation questions. Here people of drugs?) are looking for: The point I want to make here is that • Information on program implemen­ these three purposes of evaluation produce tation. (For example, the degree to information that's useful across the program which the program is operational, how acquisition cycle, either within a single similar it is across sites, whether it executive agency at different levels of conforms to people's expectations responsibility, or across executive and about it, how much it costs, how legislative branches of government. stakeholders feel about it.)

• Information on program management. (Here the typical questions are: the The Role of The degree of control there is over General Accounting Office expenditures, the qualifications and so, with this definition in hand and credentials of personnel, the way having outlined three policy purposes that resources are allocated, whether there evaluation can serve, how does the are major problems of service delivery evaluation process work? How are its or of error, fraud and abuse, etc.) effects felt in public management? Let me illustrate that process by looking first at • Ongoing information on the current how evaluation works in the legislative state of the problem addressed by the branch at the federal level, taking the program now that the program is example of the General Accounting Office. implemented. (Is the problem As you know, the GAO has the responsibility growing? Is it diminishing? Is it of assisting the United States Congress in diminishing enough so that the its legislative activities and in its oversight program is no longer needed? Is it of the executive branch. It's charged with changing in terms of its significant providing wide-ranging, independent, characteristics?) objective information as a contribution to congressional decision-making. Now this Finally, with respect to account­ may involve policy formulation (e.g., ability, the questions are again different legislating, budgeting for, and critiquing from those of the other two purposes. Here new programs), or policy execution (e.g., policymakers want: overseeing that program management and

Eleenor Chelimsky 27 Oversight in the Congress snd the U.S. Genersl Accounting Office operations are efficient>, or accountability toxicity) had not been (knowing whether or not programs and confronted.Disadvantages attributed policies have been working), or sometimes to unitary weapons had not been even all three together on the same program. established. There are seven divisions in the GAO. I direct one of these, the Program • Doctrine was inadequately developed. Evaluation and Methodology Division (PEMD), and the primary function of that • The problem of protecting civilian division is to conduct program evaluations populations hadn't even been for the Congress. The scope of our work addressed, much less solved. involves all the topical areas of the GAO, on a continuum ranging from employment, The debates on chemical weapons in health, welfare, and education through both houses of Congress that year were energy and the environment to defense. centered on our report. Senators and Three recent examples of our work Representatives cited large numbers of can show you how the Congress uses points taken from the GAO text; more than program evaluation at the national level for 100 pages of the Congressional Record were each of the three policy purposes I've been devoted to the parliamentary discussion of talking about: to judge new programs (or what should be considered acceptable policy formulation); to examine the evidence for and against binary weapons. integrity of ongoing programs (or policy After about six months of debate, the execution); and finally accountability -- to Congress decided against the Administration establish the effectiveness of a program request. about which conflicting claims may have been made. Policy Execution. Here, I think a good example is the Runaway and Homeless Policy Formulation. Here I want to Youth Program. In 1982, the Congress discuss our work on chemical warfare. In asked us to evaluate a program developed 1983, we published a program evaluation under President Carter and designed to deal assessing the executive branch request to with the immediate crisis needs of runaway break the moratorium on chemical weapons or homeless adolescents and their and to establish a new binary weapons families. ' ° The program was a very small program. 9 The Congress had asked that we one, but had come under attack from critics examine the nature, extent, and quality of who questioned whether it was really the information brought by the executive useful. Some thought its scope was too branch to support their request for new small to affect the problem. Others felt it weapons. We used a meta-evaluation or might be overstepping its authority, that is, synthesis design and our findings were that: not working within the constraints laid down by the Congress. So the requesting • Most of the assertions made by the committee wanted us to develop empirical Administration were based on belief, information about what was in fact not on empirical evidence, and were happening in the program. therefore quite possibly inaccurate. We examined the operations of 17 of the program's 69 centers, using a survey • Many advantages ascribed to binary design. Our findings were: (1) that the weapons (principally, the Bigeye bomb) program was working almost exactly as had not been demonstrated. intended, and (2) that the groups involved -­ parents, practitioners, referring agencies • Possible disadvantages of the Bigeye and the youths themselves -- were in bomb (such as problems of mixing and general agreement that the program was arming during battle, or weapons important and its services useful. As a

Elesnor Chslimsky 28 Oversight in the Congress and the U.S. General Accounting Office

result, instead of cutting the program by management in the executive branch? half as its critics had requested, the The studies I've mentioned are, of Congress increased its funds by half. course, just three examples among many of program evaluations we've performed for Accountability. A good example is our the Congress, and I've noted their specific evaluation of the effects of the 1981 uses by legislative policymakers. All three legislative changes to the AFDC public of these evaluations were brought into assistance program. Here the Congress national debates on the issues raised, and all asked us to evaluate the effects of its own three have had tangible results: funds legislative changes to the AFDC program. deleted for binary chemical weapons, funds What the Committee on Ways and Means increased for services to runaway children, wanted was the most conclusive information and improved legislation to make AFDC possible on what had happened to earner provisions both more humane and more families in the program as a result of the congruent with the findings that AFDC legislation. Here we used a combined earners did not want to remain in the evaluation approach: an interrupted program and were making great efforts to time-series design at the national level, and stay off the rolls. retrospective before-after designs involving But in addition, 1 believe these individual data collection in each of five evaluations and others like them are having urban counties. This was a complex, lengthy a general effect on the nature of public evaluation. Our most important findings deliberations. 1think this is because they were: bring neutral information to bear on some highly charged, emotional issues in national • 493,000 welfare families had been cut political debates. I'd like to point out that from the assistance rolls as a result of in each of the three cases I've mentioned the legislation. (and in others as well), our findings were not only utilized in almost agonizing detail, they • About $93 million had been cut from were also acted upon, even though they monthly AFDC outlays. were often attacked by the executive branch. 1 think the reason for this is that But these achievements wrought major attacks which are not based on contrary hardships on earners in the program: evaluation results -- but are instead built on beliefs, on anecdotes, or on war stories -­ • Although their earnings increased on just don't weight very heavily against average, they generally experienced existential, empirical findings. So when the significant income losses that they time came to look at the facts, those couldn't make up through working. attacks couldn't change the orientation of the debate that the evaluations had brought • Many of these families remained about: that is, the debate had shifted from entirely without health insurance. emotional assertions that nobody could either confirm or deny, to highly specific Yet, despite these hardships, we found that discussion about the evidence, its quality, most working recipients did not quit their and what it showed. jobs, and did not return to public assistance. As a result, it seems likely to me that The Congress then acted, on the basis over the next 10 years or so, we should be of our findings, to provide greater seeing a strong development of evaluation transitional protection -- such as health units in the legislative branches of care coverage and food stamps -- for government in this country. Here 1refer families leaving the welfare rolls. not only to the work of the GAO but also to Now what influence can we say that that of the legislative evaluation units at legislative evaluation is having on public state levels, such as the outstanding

Eles",,, Chelimsky 29 Oversight In the Congress and the U,S, General Accounting Office

organization you've developed here in transformation of the sort I'm talking about Virginia -- the Joint Legislative Audit and would leave the executive branches of this Review Commission -- and some of the country at some disadvantage, I don't think others that I know of which are springing up they're going to let that happen. under the sponsorship of the National So it seems reasonable to me to Conference of State Legislators. I think expect that the long-term impact of also that if this strong development of growing evaluation use by the legislative legislative evaluation continues, and if the branch of government in this country should work that Is done maintains its credibility be, first, more evaluation production and and its power, then we should also be seeing use in the executive branch; second, a concomitant development of evaluation in improved policy formulation, stronger our executive branch agencies, at both state program execution, and increased and federal levels. accountability in public affairs generally; Why do I think this? Well, the logic and third and perhaps most important over behind it is that, all things being equal, if the long term, greater belief on the part of program evaluation is proving itself useful Americans in the capabilities of their to legislatures across the country, then that government to address public needs success will be translated into a bigger efficiently, equitably, and with vision. legislative demand for evaluations of executive branch programs. But if the legislative use of evaluation should continue [Note: The views and opinions expressed by to rise, then the balance of analytical power the author are her own and should not be -- as between the executive. and legislative construed to be the policy or position of the branches of government -- could eventually General Accounting Office.J be transformed, unless there were a strong development of similar evaluative resources in the executive branch. But since a power (GErm mR'2 Notes (1) George F. Madaus. Scriven. Michael and Stufflebeam, Daniel L.. Evaluation Models: Viewpoints on EducBtionaJ and Human Services Evaluation. KJuwer-Nijhoff Publishing Co" Boston. 1983, page 4. (2) DId Testament, Daniell: 12-1 7, cited in David C. Hoaglin, Light, Richard J" McPeek, Bucknam. Mosteller, Frederick, and Stota. Michael A., Data for Decisions, Abt Books, 1982. page 7. (3) See the discussion by Aaron WiJdavsky in "The Political Economy of Efficiency: Cost-Benefit Analysis, Systems Analysis, and Prog'em Budgeting," in Public AdministrBtion Revi6w, December 1966, pages 293-302, and by the same author in "Rescuing Policy Analysis from PPBS" in Public Administration ReVl'ew, March/April 1969, page 190. (4) Steven E. Rhoads, "Economists and Policy Analysis:' in PUblic Adml'nistration Review, March/April 1978, pages 114; and E.S. Quade, Analysis for Public Decisions, Elsevier, New York, t975, pege 22. (5) Peter H. Roasi, Freemen, Howard E. end Wright, Sonia R., Evaluation: A Systematl'c Approach, Sage Publications, 1979, pege 24. (6) For example, Doneld T. Cempbell, "Factors Relevant to the Validity of Experiments in Social Settings,"' Psychological Bulletin, 1957, 54 pages 297-312; or "From Description to Experimentation: lntarpreting Trends as Quasi-Experiments," in C. W. Harris lEd.1. Problems in Measuring Chang8, University of Wisconsin Prasa, 1963; or, with Julian C. Stanley, "Exp8f'imental and Quasi-Experimental Designs for Research," in N, L. Gage lEd,), Handbook of Research on Teaching, Chicago, Rand McNelly, 1963. 171 See especielly, Lee J. Cronbach, "Course Improvement Through Evaluation," Teachers Col/ege Record, 64 (1963) Columbie Univ8f'sity, New YCN"k; end lwith A. L. Edwards) '"Experimental Design for Research in Psychotherapy," Journal of Clinical Psychology, Vol. 8., Jenuary 1952, pages 51-59. (8) Note the diacussiona of meesurement problems !reliability and velidity) and statistical analysis int8f'pretation in Paul F. Lazarsfald, "lnt8f'pretation of Statisticel Relations as a Reseerch Operetion" in The Language of Social Research, Free Press, Glancee, lll., 1955, pp. 115-125; and especially Lezarsfeld's evaluation of voting behavior, using a longitudinal study design, in Paul F. Lazarsfeld, 8erelson, Bernard, and Geudet, Hazel, The P9ople's Choice, Columbia University Press, New York, t948. (9) See, fCN" example, John F. Holahan's applicetion of cost-benefit techniques to evaluate a rehabilitation program in A Banefit-Cost Analysis of Project Crossroads lWashington, D.C,), National Committee fCN" Chilcken and Youth, 197D; or the General Accounting Office's recent use of an applied social research model fCN" eveluating the DOD's joint test and evaluation program lUnited States General Accounting Office: How W8J1 Do the MiHtary Services Perform Jointly in Combat: DOD's Joint Test and Evaluation Program Provides Few CrseJible Answers, PEMD-84-3). (10) Aaron WildBvsky, "The Political Economy of Efficiency: Coat-Benefit Analysis, Systams. Analysis, and Program Budgeting" in Public Administration Review, December 1966. pege 306, 111) John Evans, "Is Educational Evaluation Dying?" paper presented at the 1984 ennual meeting of the American Educational Research Association lAERA) in New Orleans as part of Sesaion No. 9,33, "Whither Evaluation~ife, Death, or Reincarnation?" (12) U.S. General Accounting Office, Chemical Warfare: Many Unanswered Questions. GAO/IPE-83-6, April 29, 1983. (13) U.S. General Accounting Office, FederaHy Supported CMters Provide Needed Swvices for Runaways and Homeless Youths, GAO/IPE-83-7. September 26, 1983. (14) U.S, Generel Accounting Office, An Evaluation of the 19B1 AFDC Changes: Initial Analysis, GAO/PEMD-84-6, April 2. 1984. (15) Auditor General of Canada, Program Evaluaton: Report of the AuditCN" General, 1983, Chapter 3, Section 3, 27, Ottawa, Ontario. Ingredients for an Effective Oversight Function

The Wisconsin Legislative Audit Bureau has about 70 nonpartisan, professional staff, and we release about 40 reports each year. In addition to doing program evaluation, we are the State's financial auditors. I am appointed by a joint bipartisan committee composed of the legislative leadership. Our reports are submitted to the Joint Audit Committee. In general, I think the slogan for Dale Cattanach program evaluators in state legislatures around the country has to be the Avis slogan, "We try harder." Program evaluation does not have the natural State Auditor constituency among legislators which some Wisconsin legislative staff functions have. Fiscal staff are important because the budget is the largest single task facing the legislature each year or each biennium. Leadership staff are there day in and day out helping in many ways. Research and substantive staff help keep their standing committees moving ahead. Therefore, evaluators have to assume responsibility for making sure their work is useful to the legislature, if they want it to be used. What are the ingredients of an effective oversight function? Several of the key factors have already been mentioned: interested and committed leadership; a favorable environment, ...... '-' including a tradition of fiscal conservatism, a strong sense of legislative independence, and an absence of partisan bickering; a strong audit or evaluation committee; and good staff. I do not intend to elaborate too much on these points, but I would like to reinforce them with one or two comments. Legislative Commitment First, the need for a commitment among the leadership to program evaluation cannot be overstated. The nature of the oversight function means that leadership

31 Ingredients for an Effective Oversight Function will, at times, have to make some hand, the authority should be restricted to a appointments which do not represent a manageable level. In Wisconsin, the legislator's first choice. Legislators do not Legislative Audit Bureau's authority is typically seek assignment to the Audit generally restricted to state agencies or Committee for several reasons. First, they agencies receiving substantial state know it's no place for those legislators who support. In this way, we are not spread thin want to be advocates for certain programs, by trying to audit every local entity. because auditors tell it like it is, not like On the other hand, for those agencies advocates might hope that it is. Second, the we audit there are few limits on what we Audit Committee deals with some very can audit within those agencies. The only controversial issues; many legislators would significant exception is within the prefer to avoid controversy or at least University of Wisconsin, where we are control which controversies they become prohibited by law from auditing issues involved in. This reluctance to serve on the involving academic freedom. Since some Audit Committee among many legislators within the University would argue that means that the support of the leadership for everything is related to academic freedom, auditing and program evaluation is crucial. we have, at times, practiced "brinks­ manship" with the University, although we Relations With Other Committees have avoided outright confrontations. Another crucial factor in the success Recently, our authority has been expanded of the program evaluation is the Audit to include a number of quasi-governmental Committee's success in working with other operations as well. committees, particularly the Appropriations Committee. Our Audit Committee receives Access to Records our reports, typically holds public hearings A second important structural on them, and may introduce legislation to requirement for effective program implement recommendations. Frequently, evaluation is access to records. We have since our reports deal with fiscal and policy only a few limits on our access to records, issues, the Audit Committee will turn over a notably individual income tax returns. I also problem with recommendations for changes have the statutory power to subpoena to our Appropriations Committee for records when necessary, though I have used inclusion in the budget. Good relations are this power only once or twice in the last essential if these recommendations are to seven years. be considered and accepted by the Appropriations Committee. The current ties between our Audit Topic Selection Committee and Appropriations Committee In addition to structural factors are excellent. The co-chairs of the affecting the success of program evaluation, Appropriations Committee this past year I believe the methods used to select were the co-chairs of the Audit Committee evaluation topics are also important. Our the year before. Five of the 16 members of evaluation topics can be (1) statutorily the Appropriations Committee are former mandated, (2) directed by the Joint Audit Audit Committee members. Our staff also Committee, or (3) initiated by staff. This is has a good relationship with the fiscal staff. a good mix. If a legislator is interested in a subject and our audit produces findings and Scope of Authority recommendations he or she finds useful, the There are at least two other structural chances of legislative follow-through are factors which are important to effective increased. I have seen situations where a program evaluation. First, the scope of the single legislator has taken an issue and audit agency's authority and responsibility almost single-handedly guided it through the must be carefully defined. On the one legislature.

Dale Cattanach 32 Ingredients frx an Effective Oversight Function

On the other hand, allowing staff the there is a substantial lack of agreement on freedom to select topics is also useful. In this point, both among legislators and staff. the course of doing one evaluation, staff We all recognize the need for autonomy, may run across another, or staff, like independence, objectivity, and legislators, sometimes have a "nose" for an nonpartisanship in our work. We recognize issue. They can anticipate what will be of the danger of "advocacy." Excessive interest to the legislature. advocacy quickly leads to loss of credibility, and that is fatal. In my view, however, there needs to Timing be a balance between independence and Another factor in success is timing. advocacy. Just as advocacy can lead to loss Not only must the evaluation report be of credibility and effectiveness -- to say there when the legislature needs it, but its nothing of the loss of job -- so too can success can be enhanced if it arrives at a excessive autonomy or independence lead to crucial moment in the debate. This year the isolation. I suspect the single biggest University of Wisconsin requested a complaint from legislative program significant amouot of fuoding for faculty evaluation staff throughout the couotry is "catch-up" pay to bring salaries in line with that legislatures, Virginia excepted, do little those of its peer institutions. In the heat of or nothing with the findings and this debate, we reported that the University recommendations in program evaluation had accumulated substantial unneeded reports. To overcome this, I think we need reserves in the accouots used to build and to: (1) work closely with interested maintain student-financed facilities, such as legislators; (2) suggest or ask for a public student uoions and dormitories. Our major hearing on our reports; (3) work with the recommendation in the student report, that media so our story is told; and (4) try to be a $22.5 million be taken from the reserve part of meetings in which legislators are accouots and used for other purposes, discussing the issues and will find our received immediate and, I think, more information useful. Representative concentrated attention than it would have Morgan's point is excellent: legitimate received if the other university issue had advocacy is professional defense of the work. not been on the legislative agenda. Beyond these factors, I believe success in program evaluation requires luck, The Question of Advocacy realism, and patience. Finally, let me say a word about the need for marketing in order to ensure the success of evaluation. It seems to me that @¥III 1m! M

Dale Cattanach 33 Ingredients for an Effective Oversight Function

We have a unique setup in South Carolina. We are the only state of the Union with an oversight organization that does not exactly report directly to the General Assembly. Our organization by law is made up of three citizens, elected by the legislature after a merit selection nomination process. George Schroeder They are elected to staggered, six-year terms. Six ex officio members, who are leadership people from the House and Executive Director Senate, also serve on the council. But only the three citizens vote on questions of audit South Carolina or on questions of personnel. They are the Legislative Audit Council only ones who vote on my appointment, for instance. These three citizens cannot have been members of the General Assembly for at least two years prior to their election to the Legislative Audit Council. And they are not paid, except for per diem expenses on the days we meet, which is usually about once a month. They always have been people who are very intensely interested in the work of the Legislative Audit Council. We have had continuity over the ten years we have been in operation in that one member has served all ten years. And we have another member who has served most of that time. In fact, our least-tenured member, who is currently our chairman, has been with us for six years. This continuity is very important, I think, in the success we've enjoyed in South Carolina. As legislators or others working directly with legislative committees and audit committees, you probably question how effective we could be under that kind of an operation. You might think we would be somewhat insulated from the legislature. That, to a degree, could have been true, but it did not work out that way. It allowed us to set up an independent, professional operation in a state where that was not the

34 Ingredients for an Effect;vs Ovars;ght Funct;on norm. And, it allowed us to sell that producing? Is it efficient and effective? organization to the legislature. They, after Does what it is doing help the general all, had put the law together that created public? us, and they'd done that because they In essence, we report to the wanted this independence. They probably taxpayers. We look at the state as a got more independence than they wanted, corporation, the taxpayers as the but I think they learned to live with it, shareholders, and the legislature as the because of the kinds of things we could do board of directors. The different for them. government agencies that we audit are really divisions inside that corporation. Proving Ourselves That's the way we see our responsibility. to the Legislature We have full subpoena authority. Early in our work we did an audit of Recently we got, through a court order, the Department of Social Services. During access to all the tax records of the tax the course of that audit we found that there commissioner in our state. That's been a was a way in South Carolina for government unique experience: to know that we have agencies to accrue funds and keep them in that authority. It just passed the legislature banks outside of the treasury. It would not in the last couple of years. You must have a be "known" in the general fund that these pretty broad authority to get at records and funds existed. Then, if the legislature voted do the type of work that we all do. to fund a program, and later Congress also voted for that program, the state agency Becoming Institutionalized could spend federal money to fund the How do our organizations become program and bank the state money in a institutionalized? How do they become private bank account. something that the legislature thinks of as We came upon this in the course of an necessary and permanent? In the early agency audit. The Audit Council had only years, I didn't know whether we were going been in existence for three years at that to continue to exist or not. We had very time, and we hadn't quite proven ourselves powerful enemies. We made very powerful to the legislature yet. The following year enemies because we audited agencies that we decided to do an audit of this problem had been adopted by certain Senators and statewide, to see what kinds of funds certain members of the House. And we existed, in what places, and so on. That crippled those agencies, and in so doing we year South Carolina ran a 16.5 million dollar put at risk our own existence. deficit in July. In September, we collected That's no longer the case in South 40 million dollars, and we were in business! Carolina. But how did we get to that For success, you also have to have a point? I think we got to that point because, very good staff. We've put a strong stress first, we have been very careful to be right on education and training with our staff. when we produce an audit. We have a very None of our staff are political appointees, in strong internal review process that we go the sense that they serve because they were through. We must have good, solid suggested to us by someone in the political documentation behind every statement we arena. In fact, a good number of them are make in an audit report, or that report will people who have come from other states, not go out. As I mentioned, we have a although the majority are from South well-trained, well-educated, and aggressive Carolina. staff. We do all the kinds of work that our sister agencies do: compliance audits, broad Ingredients for Integrity scope auditing, performance auditing. What You've got to have moral courage -­ we're trying to do, of course, is find that the courage to say what you found, no bottom line. What is the government matter what, and no matter whom it might

Georga Schroader 35 Ingredients for sn Effective Oversight Function hurt. If you don't have moral courage, if an audit report we do it at a press you try to play political games with audit conference, and we always have overflow reports, you're going to have problems. problems. Over the years we've provided Some people are not going to believe you, many good stories through our audit reports and when some people start not believing in to the media. I personally make it a part of you, they're going to convince everybody my job to work with the media on any long else not to believe in you. And in a few article or feature story they want to do. years the thing is going to deteriorate, and And they have been most helpful, I think, in you'll be gone. establishing the Audit Council, in making You can't be afraid of what people say the legislature and, most importantly, the they're going to do to you. We've had many public aware of our work, and therefore in lobbyists come in over the years and say, making sure that our work means something "I'll have you taken care of," because we and that something comes out of it. were going to bother their programs. We You've got to tell the legislature, have legislators who have publicly said, when they're wrong, that they're wrong. "That agency is going to go," meaning the When they're right, you've got to tell them Legislative Audit Council. You can't be that they're right. You've got to be honest afraid of them. You have to make them with everybody. You have to have afraid of you, eventually, and we've been integrity. If you don't have that, if you play able to do that in South Carolina. politics in this business, eventually you're You must develop good relations with going to run out of the major thing that the press. It's dangerous to deal with the really makes a legislative audit organization press, because sometimes they don't get -- and that's luck! You've got to have lots things right. They sometimes write things of good luck, and we've had very good luck up in unusual ways. But you must not be in South Carolina. critical of them. Never pick a fight with anyone who buys ink by the barrel! We try to maintain a very close relationship with the media. When we issue

G60rge Schroeder 36 Ingredients for an Effective Oversight Function

I feel there are three primary ingredients necessary for effective legislative oversight. They involve the staff, the statutory framework and the performance of the organization. Staff One of the factors we most often take William Thomson for granted, but the most critical factor of all in effective legislative oversight, is the staff assembled to do the job. The staff has Director to be professional. We see many cases in Performance Audit Division which the staff is assembled from patron­ age, part-time sources or borrowed from Office of the Auditor General other staffs. When you ask a staff to do a Arizona job that's difficult, a job that is highly specialized, you need a professional staff. Of equal importance is the talent level of that staff. This is exciting work, work that is very interesting, challenging, varied, and fulfilling. The nature of the work is such that we can and should get the best talent possible. We find we are able to attract people from the top 10% of their class; as a matter of fact, it would be unusnal for us not to get someone out of the top 10% of their class. We also find there is a growing interest in a career in this profession. When we first started our office, many of the people we got were interested in administration - being a city manager or a department head - and they saw this field as a stepping stone. We are now attracting people who have studied and had course work in program evallllltion or performance auditing; people who come out of school interested in this field as a career. They are interested in rising within the organization and/or moving to another organization doing this same work. You have to have an excellent staff and I think you can get such a staff. Another important factor is that the legislature recognize and be exposed to

37 Ingredients for en Effective Oversight Function the quality of the staff. In the past, the must also be provided adequate previous director was the primary source of opportunities for training, workshops, and contact for the legislature. They had no exposure to publications. idea of the staffing below him, and the fate There is also a need to express to staff of the division rode heavily on their reaction a sense of purpose, and a sense of to his personality, whether positive or appreciation. Legislative appreciation is negative. We try in all iustances to give our very important. We were fortunate to legislature as great an exposure as possible receive recognition from NCSL last year. to our staff so that they are aware of the After we did, we received letters of Division's talent. congratulation from the Speaker and other You not only need a talented staff, I legislative leaders. These letters have had a think you also need a certain level of very positive effect on our staff. Staff also experience in your staff. This is a need to feel they are in a professional somewhat specialized business, a business in organization, that the salary, the offices, which most of the training has to be done on the job titles, all reflect a sense of worth the job. We believe it takes from one to and professionalism. two years to train and get full productivity from a person. We rmd it takes up to three Statutory Framework years' experience before a person is able to The second primary ingredient run a project for us. We have people who involved in the effectiveness of legislative move into those positions in less time, but oversight is the statutory framework of the frankly the more experience they have the statutory powers involved in the oversight. better they are able to run the audits for us. One of the more important things that I see If we have turnover it affects both needed is statutory provisions establishing quality and productivity. We have learned independence. I'm talking about an from sad experience that when we lose independence in which we may be told the staff, even staff not considered our questions to ask but are not given the strongest staff, we see a noticeable drop in answers we are to find, the type of the amount of work that can be performed. independence in which staff reports are not We also find that if our turnover rate is too rewritten, the type in which staff have the high, our quality of work also declines. ability and professional responsibility to By quality of work, I mean the ability report based on their best professional to take on the more complicated, the more judgment. difficult issues that require more insight. In those states in which there has been Most staff can take on the more some lack of that type of independence straightforward issues, but the issues of real alleged I believe we have seen a decline in importance to the legislature, that often the effectiveness of those states' oversight takes experience to do. We also find we functions. Parties change, powers change can't replace this experience by hiring within parties, as well as within or between people from the outside. We have tried it houses. If a function is known as being with mixed results. There are simply so few political, as being part of the partisan organizations engaged in this work that process, a lack of confidence and eventually experienced people usually aren't available. a lack of use of that staff will be seen. When you get your talented and The staff needs broad powers for experienced staff, they still have to have access to confidential records. When we the tools; for iustance, one of the hottest establish oversight we are giving people a things in our field right now is computers. charge to be working in data. I see many Our experience shows computers increase cases in other states where there are productivity, increase analytical capabilities questions on access to data. What can't be and improve staff morale. If you want staff seen can't be audited. The access powers to perform, give them the tools. They have to be strong, broad, and far-reaching.

Williem Thomson 38 Ingredients for en Effective Oversight Function

We have been fortunate in our state to have performance of the staff. I am speaking of had few statutes in which we have been shut project control and management. There is a out of records. In every instance where great need for staff to be able to respond to those statutes existed we've gone to the the legislature, to meet deadIines, to be legislature, and it has revised the statutes able to project when information can be to give us that authority. provided, and then meet that commitment. We find that an important element in It is particularly critical when trying the statutory framework is how the linkage to report either ahead of or very early in a is established with the legislature. A lot has legislative session. We fowld that in some been said about Sunset. One thing good cases in the past we were not given special Sunset did do was establish a linkage with assignments from the legislature because the legislature. Sunset established a list of they did not think we could meet the time audits to be conducted (including legislative frames needed. If they perceive that an hearings) and provided that the legislature organization is too slow or not timely, they must take specific actions. will reach to other sources for information. However, there are many changes On the other hand, we were able going on in Sunset. We are now finding a earlier this year to perform the first of our different kind of linkage being established. audits in the Department of Corrections. Two sessions ago our legislature passed We were originally asked if we could do the statutory changes that require the Oversight audit in two weeks. We suggested that that Committee (which had a perfunctory job was impossible and were able to negotiate before) to prioritize which audits on the 10 weeks. Eight staff were assigned and we Sunset cycle will actually be audited. The put in about 3000 hours (with some staff Oversight Committee may also add audits to working 40 hours overtime in some weeks), the cycle or may ask for follow-ups. This and we met our commitment. We provided linkage with the legislature is going to help the legislature a service by providing the us be even more effective. We still have information; we did ourselves a service by the legislative linkage (all the statutory establishing that we could respond to requirements for hearings, etc.) legislative needs. encompassed in the Sunset statutes, plus we Another thing an organization has to are now being assigned to do work of more be concerned with in performance is interest to the legislature. producing a consistent product. I think you For instance, we are currently often hear the phrase "You can't afford to completing three audits of the Department be wrong." One mistake can discredit an of Corrections and will have issued a total entire report; one bad report can discredit a of six audits before we are done. whole audit organization. We must always Corrections is one of the most critical be right. We can never afford to be wrong. issues facing our state. The legislature is We have to be consistently right. We also extremely concerned, and I think they will need to do the same depth of research on all find these reports of far more interest and reports. use than reports on many of the obscure We can also never duck tough issues, agencies found in the Sunset law. even if we know they are going nowhere. An important statutory power needed For instance, many states can give horror in effective legislative oversight is stories of doing Sunset audits of the barbers follow-up. There is clearly a need to and cosmetologists. We took a political follow-up to the legislature as to what is beating when we stated that cosmetologists occurring after a report is out. did not need to be licensed. Four or five hearings were held, and 200-300 Performance of the Organization cosmetologists were at each hearing. The The third of my primary ingredients cosmetologists raised money, formed a for effective legislative oversight is political action committee (PAC) and hired

William Thomson 39 Ingredients for sn Effective Oversight Function the former President of the Senate to lobby insight to find the issues of concern to the for them. (A friend of mine who is the legislature. When we begin an audit, we director of the Arizona Chapter of Common attempt to obtain all legislative input Cause liked to refer to that PAC as the possible. There is often no specific mud-PAC.) The cosmetologists generated legislative reason for the audit other than a thousands of signatures on petitions and general sense that the agency needs an postcards, and the licensing of overview. (Occasionally the legislature may cosmetologists was never seriously in provide us with two or three questions.) question. Staff must be able to understand and As nearly as we can tell, however, recognize what is valuable to the legislature that report had important long-range and that which has little materiality. ramifications for us as an organization. The legislature and all concerned parties knew that we could have ducked that issue, but In summary, the three primary we did not. We took it on, knowing it would ingredients for legislative oversight are: be a political beating for us, and established The staff, the statutory framework, and that we don't pull punches and don't duck project control and management. Of these, tough issues. the staff is the most critical. If you don't The final thing on staff performance is have good staff, it does not matter what learning to find the critical issues. This else you assemble. work is unlike many other works. The scope is quite flexible, and there is often great need for staff discretion, understanding, and

Willism Thomson 40 Ingredients for an Effective Oversight Function

Legislative performance auditing In Mississippi began in 1973 when the Legislature created the Joint Legislative Committee on Performance Evall1ll.tion and Expenditure Review (PEER). The PEER statute abolished the General Legislative Investigating Committee which had John W. Turcotte performed the permanent oversight function from 1946 until 1973. Executive Director Organization and Staffing PEER is composed of five senators Joint Committee appointed by the Lieutenant Governor and on Performance Evaluation five representatives appointed by the Speaker, one each from the five U.s. and Expenditure Review Congressional districts. The Committee Mississippi State Legislature elects officers annualiy, with the chairmanship and vice-chairmanship rotating between the houses. The PEER Committee appoints an executive director who is responsibie for selecting and managing a staff of 27. Committee rules forbid individual members from engaging in any independent investigations or attempting to influence staff during projects. All committee contacts with the staff must be cleared through the director. Relationships With Budget and Standing Committees The PEER Committee actively shares its work products with the Joint Budget Committee and the Appropriations Committees. As a result of an informal agreement between the PEER Director and the Director of the Legislative Budget Office (LBO), the two offices share a central reception and message center, copying facilities, a hearing room, and freely exchange working papers. LBO staff use the PEER reference libl'ary, and PEER staff have access to LBO's extensive files on state fiscal operations. Aithough there is no formal arrangement for

41 Ingredients for an Effective Oversight Function

interaction with any standing committee, the professional competence or character of including the budget committees, it has the staff. been PEER's policy to share PEER An irresponsible executive will information and staff with all standing typically contact client groups, the news committees. PEER's staff attorney, a media, and influential public officials, and former legislative draftsman, has (with varying degrees of subtlety) accuse maintained his contacts with the drafting the evaluation staff of using "gestapo offices. PEER's publication coordinator tactics" or of being incompetent or communicates on a daily basis with the otherwise incapable of understanding agency librarians employed by the Legislative programs or management practices. What Reference Bureau. These networks keep usually provokes such contacts is an PEER alert to legislative interests and analyst's pointed questions or discovery of expedite interaction. questionable agency procedures. Contact with the standing committees I am not suggesting that performance is steadily increasing -- attributable to a audit staff do not make tactical mistakes 1983 Mississippi Supreme Court decision during projects or always have pleasing (Alexandar v State, 441 S 2d 1329) which personalities. PEER has learned to removed legislators from service as anticipate contentious scenarios and makes members of executive boards and every effort to train staff in appropriate commissions. Consequently, committee behavior. However, when members of my chairmen and senior members who once committee or influential legislators are performed direct oversight of executive contacted during projects and told of PEER operations and had immediate access to staff "atrocities," fortunately most of the executive information through service on time they do not side with the accusing key executive control boards (e.g., Budget agency. Instead, they consider the plausi­ Commission, Personnel Boa,rd, ,Building bility that such accusations are likely to Commission, Medicaid Commission, arise when auditors are keen and effective. Retirement Board, and Central Data There are other actions which are very Processing Authority) are beginning to rely supportive. I would strongly encourage more heavily on PEER's performance audit legislators to use legislative audit findings staff. during appropriation or executive PEER's philosophy on interaction is confirmation hearings as a basis of that the legislative audit function cannot be interrogation. Also, if the audit report does effective without integration with all other not contain the specific information needed legislative processes and that the function is and it is available, request that the staff a failure unless the legislative institution is provide it. Often much information is the primary beneficiary. omitted or summarized in the report while available in project working papers. Oddly enough, many legislators hesitate to ask for Support from the Legislature briefings or other assistance from our staff. and Review Committee Our staff, and I know the staffs in other speaking as a staff director who must states, will go to great lengths to make frequently assume a contentious posture audit projects useful to legislators. If a toward recalcitrant agency heads and client legislator does not have the time to read the groups, there are times when legislators can document, legislators should ask for a be most helpful. Accountability, at least in personal briefing or a shorter version of the Mississippi public administration, is still executive summary. rare. PEER members perform a vital role Topic suggestions are extremely of defending the staff when an irresponsible helpful, and the narrower the scope, the executive "circles the wagons" and attacks better.

John W, Turcotte 42 Ingredients for en Effective Oversight Function

"It takes a physician to evaluate a Other Factors public health program, an for Effective Oversight engineer to evaluate highway (1.) A well-disciplined and highly administration, a teacher to qualified staff is essential. Staff cannot evaluate education, etc. Our afford to make mistakes. Generally, the program is run by (profession X). most successful staff have strong It takes an X to evaluate our communications skills coupled with a program. The auditors were background in quantitative research simply not qualified." methods. Success often hinges on one's ability to adapt to the legislative "We were aware of the \ environment without losing objectivity and deficiencies and were in the independence. In order to establish process of correcting them before sufficient credibility, an oversight staff has the report." to perform research that is far more difficult than that which could be performed "The staff took tidbits and blew by a legislator's personal aide. Most them out of proportion, made legislators expect methodological rigor and mountains out of mole hills -- its sophistication from performance auditors. just a tempest in a teapot." (2.) Legislative audit committee members should understand that reports "Federal regulations required us cannot always be snmmarized in one to do it this way." paragraph or during a ten-minute briefing. (Please overlook the fact that PEER "There is no state money in this provides a one-paragraph synopsis on the public program. We fund it cover of our reports!) Membership on such a through user fees and the committee requires considerable reading proceeds of a trust fund. and listening. This does not mean that staff Therefore, the criticisms in the should be permitted to be obtuse, wordy, or audit were uncalled for." monotonous -- the point is that an oversight committee assignment is different from "The tone of the report is too regular legislative work, where the focus is negative. Instead of saying we more narrow and the pace, by necessity, did not 'meet' our objectives, the faster. report said we 'failed' to meet (3.) "Savvy" by legislators when our objectives. This ill editorial dealing with boilerplate executive "fend language. Why didn't the audit offs" to oversight findings tends to bring out all of the good things strengthen the process. There is a lack of the agency does?" originality in public responses offered by executive agencies to critical audit findings. Legislators should be wary of such overworked responses and require the agencies to offer better answers. Some of my favorites include:

John W. Turcotte 43 Directions for Oversight in Virginia

One of the obvious values of a conference of this kind Is that we are exposed to a whole range of perspectives. When we put them all together we have a composite picture of legislative oversight Ray D. Pethtel and its current status across the nation. It's an encouraging picture, I think. It seems to me that if there has been one single mes­ Staff Director sage underlying each presentation, it's that oversight is alive, healthy, and steadily Joint Legislative evolving to meet legislative needs. Audit and Review Commission The cornerstone of evaluation and oversight in Virginia is the systematic, of the Virginia comprehensive, and rigorous review of General Assembly agency programs. That is, much of our work is concerned with broad program areas or is --- done in a series that sweeps across performance at the program level regardless of the artificial limitations of agency boundaries. And, as a staff, we are committed to rigorous quantification and analysis of data to establish credibility and maintain objectivity. Program review and evaluation itself, of course, is not new. It is, or at least it should be, part and parcel of any program, in any field of endeavor. It has been the subject of much scholarly writing and has long been an accepted field of study in universities. But its successful implementation in our particular setting -- within state Judy Frye. of the Nation.1 Conference legislatures -- is still a recent phenomenon. of Stat. Legislatur.., p,.....,t. Ray P.tht" with For it to work and endure in this setting, we an .w...d for hi. profaaaional contribution. have had to leam to deal with challenges to legilJatlve program evaluation. that weren't always foreseen in the textbooks. Those are the realities that inevitably come into play when dealing with public policy considerations in the legislature. As we came head to head with the enormous range and diversity of government programs and the intricacies of the political arena, some things became clear to us as staff. We had to explore and devise

44 Directions for Ovsrsight in Virglnls new linkages to mesh professional, objective long-term care in nursing homes, and the staff research with the volatile, political certificate of need law. Those reports have environment -- without compromising served as a good part of the health plAnning either partner. We had to learn how to use base now in place in Virginia. Five addi­ a wide range of sophisticated and technical tional projects were completed within the methodologies, yet make sure we translated Individual and Family Services functional our findings into a language that the area of state government. One report in legislature found useful. We had to recruit that series was a rigorous assessment of the staff with a high degree of analytical ability quality of resident life in the State's homes and then give them the precise kind of for adults. Our recommendations were used program and policy exposure that made to make major legislative changes to fund them prime candidates for jobs in the adult home care, to improve some executive branch And we had to develop deplorable conditions, and to enhance costly in-house resources for data State-level supervision. processing and computer analysis. Seven projects were completed within I like to think that in Virginia we have the Transportation function. Reports been successful in each of these areas. In ranged from an evaluation of the organi­ large part, our success has been due to the zational structure of the Department of support given this enterprise by the leader­ Highways and Transportation to ship of the General Assembly and by the consideration of highway financing needs members of the Commission. We must also based on requirements for new construction credit the vision of the General Assembly in and maintenance. Our studies and designing our enabling legislation and the recommendations on vehicle tax equity and Legislative Program Review and Evaluation highway construction needs were used Act. extensively by the legislature in considering two major motor fuel tax bills that produced over $150 million in new revenue. An Accomplishments unanticipated outcome of that series, but a of the Evaluation Act most significant one, was a mandate for us When the Evaluation Act was being to examine the equity of highway drafted some eight years ago, the com­ construction and maintenance fund mittee wanted to promote four primary distribution -- a very technical staff study goals: to expand the evaluation of public that resulted in a significant legislative programs and agencies using existing overhaul of the existing distribution statutes oversight processes, to encourage greater and formulas. participation by standing committees, to Two projects were in the Resources ensure systematic scheduling of evaluations, and Economic development function. Those and to encourage greater use of evaluation studies reviewed the performance of 29 findings. regulatory boards and assessed the regu­ This Conference was required by latory system as a whole. Those reports statute as one means of gauging how well continue to serve as a primary information those goals were accomplished. In making base for the modernization of our regulatory that assessment, I think it's helpful to set system. forth the record and look at a few cumu­ Fourteen projects were incorporated lative statistics. under the General Government function. A lot of evaluation has been carried One of those projects assessed the structure out. Forty-one projects have been com­ of the executive branch of State pleted or are in process under the Act. Five government and resulted in a substantial health related projects were completed overhaul of cabinet-level and agency-level under the initial pilot provisions. They dealt organization, and devised a rational system with in-patient and out-patient health care, of agency and board classifications.

Rsy D. Pethtel 45 Directions for Oversight in Virginis

Seven projects have been completed or legislators have worked with JLARC on are in process in the Administration of various subcommittees. Of those, 52 served Justice function dealing specifically with on projects directly associated with the the State's correctional system. Topics Evaluation Act -- 29 members of the House range from security procedure and staff of Delegates, and 23 members of the needs at major prisons to inmate projections Senate. Seventeen legislators served on two for the next decade. One ongoing study or more projects. deals with the capacity and population of Turning to the record on utilization, I the State's local jails -- a matter of refer you to our 1985 Report to the General considerable controversy that should Assembly, which has been prepared to bring capture legislative attention. that record to your attention. In general Finally, we are beginning an education terms I can say our reports have been series which is to focus first on Elementary heavily used. Many public hearings have and Secondary Education and then on Higher been held on reports and on specific Education. Our first assignment is to recommendations. Oversight findings have prepare an objective and rigorous assess­ been discussed at numerous committee ment of what it will cost to fully fund the meetings during the last seven legislative State's share of the educational standards of sessions. Dozens of bills have been passed quality -- a contingency that has been by the General Assembly and signed into law estimated by the education department to by the Governor, and literally hundreds of require over $515 million in new state administrative recommendations have been appropriations. adopted by executive agencies as a result of These projects have comprised 75% of legislative endorsement of our reports. the workload of JLARC, and as you can see In somewhat more dramatic terms, of many of them are the "big ticket" items the $166 million we can identify as that have been squarely on the legislature's one-time savings or new revenues achieved high priority agenda. as a result of the adoption of With regard to the goal of increasing recommendations contained in our reports, the interaction between JLARC and the $126 million can be directly attributed to standing committees, the record is equally studies carried out under the Evaluation good. In addition to the statutory linkages Act. JLARC has with the House Appropriations I believe that's a record of and Senate Finance Committees, many accomplishment that would be judged very important ad hoc linkages have been acceptable by the members who drafted the establish with the standing committees. Evaluation Act. It is a record that the These include coordinated work efforts with General Assembly can take pride in. the House of Delegates committees on Health, Welfare, and Institutions; Roads and Internal Navigation, Counties, Cities, and Ongoing Review Towns; General Laws; and Finance; and the of the Evaluation Act Senate committees on Education and But those statistics alone do not Health; Social Services and Rehabilitation; constitute our full assessment of the Transportation; General Laws; and Local accomplishments of the Act. In fact, the Government. members who designed this statutory In terms of legislator participation, vehicle -- some of whom are here with us the numbers speak for themselves. Over the today -- required that the Act receive years, a total of 17 legislators have served regular review. And the Act has been on the Commission itself. Prior to adoption assessed almost continuously since its of the Evaluation Act, there was little passage in 1978. interaction with other members. Since A special study committee reviewed 1978, however, 70 other individual the accomplishments of our first series of

Ray D. Pathtal 46 Directions for Oversight in Virginia studies in the health care area in 1979. As a be amended to accomplish an important result of that fonnal critique, a legislative objective for staff -- to improve our ability committee concluded that evaluation to plan our work on a longer-range basis. procedures were sound, but recommended Staff work planning needs to be done for at several refinements. Changes were made in least a two-year period, possibly longer. how topics were selected for review, in the This will help in scheduling staff amount of time agencies were given to assignments and in recruiting and training review draft reports, in report tone and staff with relevant skills. It will also format, and in report follow-up. provide sufficient lead time to establish in addition to that "in-house" meaningful contact with the standing assessment, we participated in a broader committees. Given current demands on the comparative assessment conducted by the time of Virginia's citizen legislators, a Eagleton institute under the direction of Dr. longer lead time is necessary. Alan Rosenthal. According to the criteria Here's how the process currently he used to measure success, Dr. Rosenthal works. The Code of Virginia directs the concluded that the JLARC model has Commission to prepare periodic evaluation worked very well. schedules. The statute states that the Further assessment is accomplished functional areas of State government will be periodically through the status-of-action scheduled for review and evaluation on a reports required of agencies. These status seven-year cycle, and that from time to reports permit us to assess on a continuing time as required, the Senate and House of basis whether 01' not agencies have taken Delegates will establish the schedule by action on report recommendatious and joint resolution. legislative direction. For the most pact, this provision of Taken as a whole, these multiple the Act has been carried out. However, assessments clearly indicate to me that the neither the Commission nOlO the General Act has worked well. Virginia has in place a Assembly deemed it necessary to schedule viable process for legislative oversight. The the functional areas on a full, seven-year metal of that process bas been tested over cycle. in 1979, a majority of the the course of several years. It is enduring, Commission believed it was not in the and it has fulfilled the expectatious of those legislature's best interest to establish a who proposed the Evaluation Act in 1978. long-range schedule, because it would provide agencies advance notice that oversight was scheduled. Cousistent with Proposed Refinements that belief, the content and specificity of to the Evaluation Act each scheduling resolution has varied, thus This is not to say that there is no room providing the Commission greater for improvement. After eight years of scheduling flexibility. For example, since experience, we have identified several areas the legislature has had a continuing interest where refinements can be suggested. These in transportation issues and programs which areas are outlined in the final article of the required nearly four years of staff work, 1985 Report to the General Assembly in what was originally conceived as a seven­ detail. Each suggested change would year cycle will now take nine 01' ten years to recognize that the legislature often uses a complete. variety of mechanisms to accomplish the But two additionallessous have been same objectives intended by provisious of learned as a result of experience. First, the Evaluation Act. I will comment briefly while the long-range schedule may signal on the more important ones. the functional area for review, it does not First, some refinement can be signal the specific topics. Thus, earlier suggested with regard to how work projects concerns about signaling agencies about are scheduled. I believe this provision can impending review are largely moot. Second,

Ray D. Pathte' 47 Directions for Oversight in Vi,ginis a longer schedule (as actually was adopted Although the Evaluation Act did not in the last resolution) has helped staff mandate program terminations as do typical better plan and sequence work and balance "sunset" laws because of the unreasonable that work under the Evaluation Act with time demands that would place on Virginia's other oversight mandates. There seems to citizen legislators, it does have a hearing be little reason, therefore, not to adopt a requirement as an action forcing device. schedule for the full cycle of functional The statute requires that a standing areas in a single resolution. committee hold hearings on reports A second area in which we suggest a transmitted to them within 120 days of the change is in the mechanism used to specify transmittal date. evaluation topics. An important objective To a great extent, the hearing of the original committee in drafting the provision has been used -- but not in the Evaluation Act was to increase the way originally conceived. Public hearings participation of standing committees in have been held on practically every report identifying topics for evaluation. But the issued under the Evaluation Act. For Act envisioned ouly one mechanism for example, public hearings were held on each topic selection, and the General Assembly" separate report of the health pilot series. has used many. Staff presentations were made at a specially In several instances, JLARC has been convened hearing, and interested agencies directed to make specific studies by and citizens commented on report findings statute. JLARC is required by law to make and recommendations. periodic performance reviews of 29 A single hearing was held on the series occupational and professional regulatory of reports on Individual and Family boards, for example. And when the Services. Dealing with several reports at Commission began its work in the functional once reduced the time demands on legis­ area dealing with the Department of lators and kept hearing costs to a minimum. Corrections, the legislature us'ed still The committees that received the reports another variation to specify evaluation on the Division for Children and the Division topics. In that case, the specific subjects to of Volunteerism each held separate hearings be studied were listed in the Appropriations which focused specifically on staff Act. And even though Transportation topics recommendations contained in the reports were specified by resolution, mandates and on the issue of whether or not to contained in the Appropriations Act "sunset" those agencies. A package of supplemented our study agenda. legislation resulted from each hearing which My point here is that the legislature advanced recommendations contained in the has a number of ways to specify the kinds of reports. studies it wants and the questions it wants The hearing provision was addressed. Thus, while the Evaluation Act implemented somewhat differently for the offers one mechanism to use for that reports in the Transportation series. In that purpose, a variety of other mechanisms are case, the reports were made available just also used. Given that several alternatives prior to the legislative session. Two major are used, and each achieves the objective of items of motor fuel tax legislation, Senate expanding legislative oversight, the General Bill 99 and House Bill 532, had been Assembly may wish to modify this specific introduced and were under consideration. provision to recognize the alternative The substance of those reports was practices. conveyed to House and Senate budget and A third amendment that will be finance committees concurrent with suggested recognizes that the General hearings on the tax bills. The House Assembly uses oversight studies based on its Finance Committee also held a hearing on interest in a subject and consistent with its the report dealing with the administration own work schedule. of the Department of Highways and

Ray D, Pathtaf 48 Direction. for Ovsr.ight in Virgini.

Transportation. Since the substance of each to be addressed. The law will terminate on of the reports had been heard extensively by July 1, 1987 unless reestablished by prior the members of the legislature (and act of the General Assembly. Because of its legislation had been passed dealing with accomplishments, I trust it will be the each report) further hearings would have pleasure of the General Assembly to reenact been duplicative. the statute at the next legislative session. In a similar way, hearings were held on the corrections reports during the legis­ Renewing the Commitment lative session; because the reports were issued just prior to the session and to Legislative Oversight legislation relevant to the reports was under In closing my portion of this panel, I consideration. would like to observe that the Evaluation Thus, the timing of a report and the Act has provided the means and the conditions associated with the subject are momentum, that has kept Virginia on the more important to report utilization than cutting edge of the legislative oversight in the public hearing requirement. Those this country. As we move into the future, it reports that are issued before the is my belief that we can continue to hone legislature convenes are frequently heard as and sharpen that edge through periodic a routine part of the legislative process. assessments. This conference has provided Reports that are released during the interim us a forum to review our goals, to measure may require special hearings. what we have achieved, and to reflect on In reenacting the Evaluation Act, where we wish to go. By reenacting the therefore, we believe it is appropriate to Evaluation Act, we will renew our make the hearing provision permissive commitment to an aggressive legislative rather than mandatory and recognize that oversight process for the future. And when report utilization is a result of legislative that occurs, this conference can be viewed commitment to the oversight process -- not not as the ending of a process, but as a because of either a "sunset" provision or a beginning. "publiC hearing" requirement. As a last item in this review of the Evaluation Act, I must also note that the sunset provision contained in the Act needs

Rsy D. Pethte' 49 Directions for Oversight in Virginia

I believe that we have had an exceptional conference. We have learned a lot about the status of legislative oversight and sunset in the states, the need for legislative oversight, what legislators expect from their staffs, oversight in the Congress, and what is needed for an effective oversight function.

L. Cleaves Manning The General Assembly's System of Oversight In reflecting on the conference Delegate, proceedings and on my 18 years as a Virginia General Assembly member of the General Assembly, I am convinced that we have one of the finest Chairman, Joint Legislative systems of legislative oversight in the Audit and Review Commission country. Let me clarify what I mean by system of oversight. We have established: ...... ,--""", • the Division of Legislative Services to provide much needed staff assistance to committees and members.

• the Appropriations and Senate Finance Committee staffs to provide the professional expertise necessary to examine the State's $16 billion biennial budget.

• the Auditor of Public Accounts' office to test the adequacy of internal controls, the appropriateness of computer systems, and the integrity of financial records.

• the Division of Legislative Automated Systems to maintain an innovative and up-to-date, computer-supported legislative information and bill drafting system.

• JLARC, which was created in 1974 and has become an important part of this comprehensive system of legislative oversight.

50 Dirsctions far Oversight in Virginia

I would like to look hack at my 12 Assembly documents over $160 million in years on the Commission and share with you savings and new revenues resulting from what I think are some of the most JLARC recommendations since the significant accomplishments of JLARC. I Commission's inception. would also like to offer a few comments about the direction legislative oversight These are just a few of the many may take in Virginia to ensure its continued accomplishments of JLARC. These success. accomplishments would not be possible without the support of the General JLARC's Accomplishments Assembly and its leadership. Informing Legislators About the Workings and Business of State Future Directions and Needs Government. Since the creation of JLARC, The future success of JLARC depends over 100 reports have been released by the on a number of factors. Let me just Commission and submitted to the members highlight a few. of the General Assembly for action. An important objective of JLARC is to collect, The Independence of JLARC. evaluate, and report information and JLARC's strength is its independence from recommendations that can be used in the political activities and pressures of the legislative decisionmaking. Reports provide legislative process. The only way to information that may be useful to gnarantee JLARC's credibility as a legislators during deliberations on fact-finding organization is to keep politics legislation, during committee hearings, and out of the study process. Factual material in responding to constituent questions or must remain objective and non-political. requests for assistance. There is a lot of room for politics after the staff findings and recommendations have Ensuring Compliance With Legislative been released. Intent. Writing and enacting legislation is the lawmaking function of the General A Study Agenda Which Relates to the Assembly. This establishes legislative Business of the General Assembly. Those of intent. JLARC has helped ensure that laws us who serve in the legislature know that are being carried out as the legislature the greatest problem is not doing intended. JLARC has informed the evaluations but using them. JLARC's legislature on several occasions when intent record of report utilization has been very was not clearly understood by program high, especially since the enactment of the administrators and when statements of Legislative Program Review and Evaluation intent seemed to have been ignored. Act in 1978. To have an audience with the General Improving Agency Efficiency and Assembly, with its committees, and with its Effectiveness. JLARC is required by members, the Commission must have a statute to make recommendations on ways study agenda which relates to the important State agencies can operate more efficiently business of the legislature. You can and effectively. Many significant changes evaluate education all you want, for have been made in agency efficiency and example but unless the General Assembly is effectiveness in response to JLARC called upon to make decisions for which the recommendations. evaluation is relevant, the study will be ruled out as not germane to its business. Saving the Commonwealth Money. To ensure report utilization, study From a legislator's standpoint, cost savings priorities of JLARC can be better are the most visible outcome of JLARC integrated with the study needs of the studies. The 1985 Report to the General General Assembly in several ways. First, I

L. Clesves Manning 51 Directions for Oversight in Virginia believe the Legislative Program Review and • The House Appropriations and Senate Evaluation Act should be reenacted. It has Finance Committees frequently direct provided JLARC with a means for JLARC to study agencies and scheduling its work and for coordinating its programs that receive a great deal of studies with the standing committees of the interest during budget deliberations. General Assembly. JLARC staff are periodically asked to Second, I believe that the General brief the committees at the conclusion Assembly should provide JLARC with a of studies. longer-term (but flexible) study plan -- at least four years. Standing committees with • Technical support to special study task jurisdiction in a functional area should be forces and study commissions like the involved in topic selection. recently created Commission on Third, I believe that JLARC should Deinstitutionalization. continue to be assigned "big ticket" study topics that have a high degree of legislative • Joint subcommittees like the ones and public interest as well as more routine established to review the Division for oversight topics. Recent "big ticket" Children and Division of Volunteerism. studies have included highway funding, correctional staffing, and educational Adequate Time and Resources for the standards of quality. JLARC is an ideal Staff to Get the Job Done. The General forum to study such politically sensitive Assembly has been very supportive of areas because of the staff's objectivity and JLARC studies over the years. In the rigorous research methodology, and because future, the General Assembly will have to the Commission has no particular advocacy continue to provide JLARC with an role in the legislature -- except to advocate appropriate level of resources to get the job legislative oversight. done properly. Timing and scheduling of projects is a Linkages with the Standing critical factor in using the reports produced Committees. An important objective of the by JLARC. Because of the complexity of Evaluation Act is to coordinate the JLARC studies, the General Assembly Commission's topic selection and report should provide staff with the necessary time development activities with the standing to adequately plan and research topics. committees of the General Assembly. By Evaluations of the State's highway programs involving the standing committees in and educational standards of quality cannot selecting and constructing study topics, and be accomplished in weeks or months. They by coordination with committees, specific may take a year or more because of the legislators who are the logical users of technical complexity associated with the oversight information have most direct topics. When legislators are asked to make access to it. Here are some examples of an important public policy decision that linkages that have worked well in the past: affects all areas and citizens of the Commonwealth, they should have • Substantive committee involvement in confidence in the information that is given selecting topics for review and in to them. making recommendations. Senate One of the most dramatic changes in Transportation and House Roads and the legislature over the past 10 years has Internal Navigation were frequently been the introduction of computers to called upon to react to JLARC expedite legislative business. We rely on findings and recommendations related computers to draft bills, to inform the to highway programs and financing. public of our actions, to prepare and analyze

L. Cleaves Manning 52 Directions for Oversight in Virginia the budget, and to write letters to executive agencies for program operations. constituents. In recent years, JLARC staff Finally, we will have to continue to have made extensive use of computers to offer political support so that we can analyze the State's highway and education continue to attract talented and skilled funding formulas. professionals trained in the areas of What used to be impossible to analyze evaluation and public policy research. r no matter how much time you had available believe that the General Assembly has set now can be accomplished. In some cases, the pace in the last 10 years in attracting data that may have taken months to analyze some of the most able public servants in the can now be assessed in hours -- given Commonwealth. In the last four years, appropriate data collection and storage. JLARC has become a supplier of top-level Legislators were able to assess the impact staff for the executive branch of of changes in the road funding formula on government in Virginia. The Governor has their districts in a matter of a few hours. drawn heavily on JLARC, the House Computers make our job easier and Appropriations Committee, and Senate provide precise information for decision­ Finance Committee for cabinet-level staff. making. But they also cost money. The r guess we are doing something right over General Assembly will have to support here. Let's continue to do it. JLARC's growing computer needs for oversight purposes if it wants to match the sophisticated technology being used by fGbml J)IJr £03

L. Cleaves Manning 53 Professional and Ethical Concerns In Legislative Program Evaluation

I would like to thank JLARC fol' the opportunity to participate in your activities and discussions about what I considel' to be a, if not the, majol' issue fol' all ol'gan­ izations in the public sectOl', as well as in the pl'ivate sectol': professional and wOl'k ethics. In recent yeal'S we have given impol'­ tance to the improvement of productivity in Jacques Goyer our ol'ganization. But the emel'gence of ethics issues now foroes us to return to basics and l'eexamine the reasons fol' the Principal existence of institutions such as yours and mine, the way we approach the wOl'k we're Office of the asked to periol'm, and what we expect of Auditor General of Canada people WOl'king in our ol'ganization. You will agree with me that this is indeed - sel'ious business, and I sincerely hope that my modest contl'ibution will be of some assistance to you. Fol' the sake of clarity, I've divided my presentation into three parts. Because I suspect that most of you are not familial' with the Office of the Auditol' Geneml of Canada, I will fil'st descl'ibe the roles and responsibilities of the Auditol' Genel'al and give you some othel' key charactel'istics to enable you to undeI'Stand the context in which our Code of Pro­ fessional Conduct was developed. The second part of my presentation will deal with the conception, development, and application of the Code. I will conclude with my own peI'Sonal views as to the majol' issues faced by the audit office and theil' potential impact on the Code. The Office of the Auditor General The Office of the Auditol' Geneml has been in existence fol' more than 107 yeal'S, and fol' most of those yeal'S, its main responsibility has been to post-audit expenditures fol' compliance and to express

54 Professionsl snd Ethicsl Concerns in Legis/stivs Progrsm Evslustion an opinion on financial statements of the The Office reports annually, that is, government of Canada. we have one report to Parliament. We Back in the late 19505 and early 1960s, report through the Public Accounts we started to investigate what were then Committee of the House of Commons, called "non-productive payments". These which is somewhat akin to the House of were transactions that conformed to Representatives in the U.S. Congress. The legislation and regulation, but provided no Public Accounts Committee is composed of apparent benefit to taxpayers. Because this elected members of Parliament from all activity was not formally in the mandate of parties, and is chaired by members of the the office and because, of course, the opposition. So it is not the government government questioned the propriety of member who sits and chairs the Public reporting such transactions, a committee of Accounts Committee, but a member of the legal and accounting experts (referred to as opposition. the Wilson Committee) was set up to The Auditor General is appointed for a examine the role and responsibilities of the ten-year (or age 65, whichever comes first) office. Recommendations of the Wilson non-renewable term of office. We have a Committee were for the most part staff of approximately 600 people, 80 incorporated in legislation which was percent in the capital, Ottawa. Some of our enacted in 1977. staff, including professionals, are covered According to the Auditor General's by collective bargaining. The office has a Act of 1977, the Auditor General is now high profile, and very often the media will required, among other things, to call to the refer to us as a "friend of the taxpayer." attention of the Canadian Parliament cases or instances where money has been ex­ pended without due regard to economy or Reasons for Developing efficiency, or where "satisfactory a Code of Conduct procedures" have not been established to You may wonder why after 104 years a measure and report on the effectiveness of code was deemed to be required. We programs. managed for 104 without anything in Now what does that mean in plain writing, and suddenly we had to have it. English? That means that we do not do Basically there were three reasons. program evaluations. We evaluate measures The first reason was related to the in place to measure program effectiveness. change in mandate. You have to understand That's a subtlety, but a very important one. that before 1975-77, the office was a The assessment of whether value has relatively homogeneous and stable been obtained for money spent, combined organization. Staff members were for the with the more traditional audit respon­ most part accountants by profession and sibility led to what is currently known in auditors by training, and their careers were Canada as "comprehensive auditing". spent in the office. in this sheltered The Office of the Auditor General is environment, professional and work values part of the legislative branch and is not a were well defined, known to all, accepted part of the government. Safeguards have and adhered to, and there was no need to been included by Parliament through put them in writing. New staff members legislation to ensure that the office is free had no choice: they either had to adapt or of government control, notably in the areas they would be ejected from the office, of budget approval and allocation, and literally. human resources management. For From 1977 on, the office had to example, classification and hiring is recruit a large number of staff members to independent from government policies and carry out the new responsibilities. Most of procedures. these new staff members were non-

JacqUBS Goyer 55 Professional and Ethicsl Concerns in Legislative Program Evaluation accountants and had little exposure to I don't want to give you the impression auditing. We are talking here about that we had a lot of those problems. They management experts, lawyers, economists, were irritants, like the fly that is six inches engineers, computer specialists, and human from your face, even though there is a resource specialists. In order to manage all thousand cubic feet of space in the room. these people, we also needed a new breed of And basically managers, the seniors ones, managers. The net result was an began to worry about the increase in them organization more than twice its original and lack of guidelines. They worried even size, managed by people coming from the more seriously about the potential public or private sector but with absolutely implications and impacts of such incidents no career roots in the office. on the creditability of the office as a During the same period the office whole. What should we do? Should we be started to use contract personnel heavily. reactive, proactive, should we continue to About 30 percent of our budget was rely to the judgement of our employees? allocated to contract personnel. You can That was the first reason, and definitely a understand what this massive transfusion of key one. new blood did to destabilize some of the The second one, also important, was work and professional values of an organ­ the appointment of a new Auditor General ization. And it led to an increase in the and a change in tenure. Before 1977, the number of real or potential conflicts of Auditor General was appointed for an interest and other questionable conduct. indefinite period of time. It was not rare to To give you some examples: see an Auditor General stay at the helm for 15 or 20 years. This was an additional •A Department tells the Audit Office: element of stability. As I mentioned "Mr. Auditor, we have a consultant earlier, legislation has changed that. In doing some audit work on our com­ 1980, the Auditor General at the time had puter systems. By the way did you to step down and was replaced by the know that his firm just obtained a current Auditor General, Mr. Kenneth Dye. contract to do improvements to the I don't have to tell you the impact of a same computer system?" Answer: change in the chief executive officer in a "No I didn't know that." large corporation. Now imagine that impact on a very small organization, highly cen­ • "Mr. Auditor, I'm XYZ from ac­ tralized, particularly when there is no limit counting firm S0&5o. Do you know to his authority. Mr. Dye did rock the boat, that one of our former employees now and starting asking a lot of questions. And working for you bas a business on the we did not have the answers to those side and, by the way, is taking clients, questions. I'll come to that a little bit and is charging low rates? We think it later. is absolutely disgusting, unfair, and we The third element, which was may go to the profession for a ruling somewhat more minor, was the inadequacies on that." of former guidelines and policies. The Code was not our first effort to crystalize them. •A preliminary audit report is released There were some elements before, but they inadvertently by a new employee were contained in various documents, were before it gets cleared by the office. difficult to access or apply, and were not You can imagine the embarrassment. specific. Furthermore, some of the policies and guidelines were applicable only to a •A management consultant now very specific group of employees. working for the office in charge of an For example, the Canadian Criminal audit is considering entering into a Code contained references to breach of contract with his former firm. Should trust by public officials and public servants. he be allowed to proceed? Our Public Services Employment Jacques Goyer 56 Professional and EthicBl Concerns in Legislative Program Evaluation

Act contained restrictions on the political to indicate to Parliament, to departments, activities of public servants. Our audit and to taxpayers the standards that govern manual certainly had a few paragraphs on the work that we do and the behavior we the need for objectivity and due care. expect from our employees. And the higher Members of the accounting profession were the visibility, the greater the need. subjected to their own profession's code of This line of reasoning had two im­ ethics, but the new employees were not, portant implications for the development of because they were not truly traditional the Code. First, we had to identify all the professionals. different actors and players in the audit Conflict of interest guidelines applied game, and we had to write the Code using to executives, but not to other employees. their point of view as a perspective. The Such fragmentation and the lack of problem became one of finding out what specificity led naturally to ignorance, they expected of us rather than what we confusion, and lack of credence. It was for thought we should be doing. those reasons that office decided to take Secondly, we were led to believe that the initiative, and I, Director of Personnel because the stakes are so high, employees at the time, was given the responsibility of cannot be left alone in making the decision trying to develop an appropriate code of as to whether they are in a real, apparent, ethics. I was assisted in that task by or potential conflict of interest, or when colleagues in the personnel division and an they run into unethical behavior. advisory committee composed of For this reason you'll find that the professionals and managers of our office. Code now requires from our employees and other personnel disclosure of some of their activities. For example, we require Developing the Code disclosure of any secondary employment; As to the development of the code, we same thing for previous work experience. started with a review of literature of all the The underlying principle is that while the cases on unethical conduct or behavior that onus is on the employee to disclose a we were aware of. We soon concluded that situation where the problem might occur, the core of the question -- and the key issue the responsibility for making the decision as -- was the question of trust. to what effectively constitutes inappro­ What I mean by this is that organ­ priate behavior or a real conflict of interest izations such as yours or mine will not be rests squarely with the office. able to exist or perform a useful function if This was confirmed by our finding that legislators, taxpayers, and others do not when employees are requested to divulge trust or have confidence in what we do or situations where they think they might be in what we say. Since trust can only be a conflict of interest, they often fail to do earned, and can never be taken for granted, so. The reason is that they genuinely we felt that it was of the utmost im­ believe that their conduct is well within portance that we protect that treasure. acceptable norms, and they simply do not We became convinced that it is not see the problem. It is others who see a sufficient to have highly competent people. problem. I found that even the most blatant The highest standard of conduct is also conflict of interest situation was very often required because, in the final analysis, the not seen by the person who had two feet in decision as to how well we are fulfilling or it. We also had to recognize that discharging the public trust that is bestowed management has the responsibility of on us belongs to those who are on the informing employees as to the values and receiving end of our work. attitudes expected. This led us to the conclusion that we We also faced another problem, the needed a code not because we were having impossibility of identifying and describing problems, but because we had an obligation all the situations, conditions, and circum-

Jacques Goyer 57 Professional and Ethical Concarns in LafJislatillB Program Evaluation stances where inappropriate behavior might the mix, the cement that put the pieces occur. We started listing the dos and don'ts, together, and there was a lot of that. I'll and it didn't work. Every time we had all give you some examples. the angles covered, or we thought we had, someone would come and point out (1) Who are the clients? When he something that wasn't covered. joined, Mr. Dye was coming from the So we had to revert to another private sector. He said that he approach, and that was a positive approach, recognized that Parliament is our recognizing there was some risk involved in client with a capital "e," but this. Instead of describing what not to do, departments also are clients. They we would identify the basic principle behind are small "c" clients. We started with a "pro-ethical" behavior. By doing so, we that, but it didn't work. There was would lose some precision, but the coverage not agreement on this. In the Code we would be better. used a rather unsatisfactory expression called "audit entities," which is much less controversial. The Basic Principals The two fundamental principles were (2) Another debate, and it was a determined to be professionalism and major one, was over the question of integrity. I am not talking about pro­ balanced reporting and construc­ fessionalism in the traditional sense. For tiveness. Some people in our office us, professionalism means we expect all our believe that implicit in the notion of employees to behave like professionals, to comprehensive auditing is the notion be competent, to look for excellence, and at of balanced reporting and overall the same time to be responsible for their assessment. Their view is that, after self-development. These two principles, all, when you go and see the doctor for professionalism and integrity, constitute the a physical, you expect him not only to rock basis on which the whole code rests. repor1 deficiencies but to give you an While working on the code, we found assessment of your health: that the main benefit and the most inter­ "You have two big warts," he esting byproduct of the code was not the might say, "and they are big and when document produced. It was the discussion I look through the magnifying glass." that took place, at the advisory committee "Yes, doctor, but what about my heart level first, and then at the executive and my lungs?" "Well I don't know committee. Even on what we thought were about that, but these warts you should very simple and safe issues, it was just like looked after!" an iceberg: we could see the visible parts, but we knew that underneath there was a lot The problem, according to others, is more. that comprehensive auditing is not defined If you look at the Code, it is only 18 in legislation. What we are in fact pages. You should have seen the drafts! requested to do is report on deficiencies. If You should have seen the discussions! I you read our mandate, we have to report on think the Code now represents a consensus cases where there has been no regard for -- I would say in some cases a compromise the value of money, where public money has reached at the end. What is not seen is the not been fully accounted for, and so on. It's process, and I think in these cases it is very negative if you read the legislation. probably as important, if not more The debate heated constantly over the important, than what you see as a result. question of recommendations. There is a If your organization wanted to take view that because auditors can only make our Code and try to apply it, I think it very general recommendations, they are probably wouldn't work, because you'd miss considered not very helpful by management,

Jacquss Goysr 58 Professional and Ethical Conearns in Lagis/ativa Program Evaluation and therefore maybe they should restrict could jeopardize or call into question their their activities to recording deficiencies. judgement or the work of the office, and it Of course, others would express the opposite also requests that they disclose any future view, and in the Code we have been unable situation that might lead to criticism. The to find a very precise way of looking at that Auditor General, assisted by the executive question. committee, is the ultimate decision maker, Departments were clear about it: and if employees don't agree with that, they "Listen, you either make intelligent can go through normal grievance procedures. recommendations, or you shut up!" But if you start making recommendations that are very precise, then you are in conflict of Does It Work? interest when you go back and audit. Three years after its publication, I can Overall, we have to say that our Code say that generally speaking the Code has of Conduct is far from being perfect. But it permitted us to solve the problems brought represents considerable effort from our to our attention. Employees have benefited people to try to achieve a reasonable considerably from the application. For consensus as to the essence of our work and example, the office decided to pay the basic principles governing the way it professional membership fees. This was should be carried out. From that point of decided on two bases. First, we wanted view, I think it's a major achievement. them to remain members in good standing with their professional associations, and second, we felt that we could control Major Components of the Code moonlighting because employees are forced As it now stands the Code has four to disclose. So that was one advantage. major components, in addition to the two The Code has also facilitated the principles mentioned before: (1) We have a career development of some of our staff, responsibility to Parliament and must because now departments and agencies are operate within the Auditor General's ready to take on some of our employees. mandate, with impartiality and with due They know the standards that we apply. regard to economy and efficiency and They know that if we send an employee to effectiveness. (2) We also have a learn about how government operates, we responsibility to audit entities: to carry out won't use that against them. It's in our audit work competently, with objectivity, in Code. a constructive perspective, with concern We have some difficulty with about confidentially and the need for implementation of the clause calling for substantiation. (3) We have a responsibility disclosure of contract work in audit entities, to the public, which is basically because there is no accepted definition of communication to the public. And (4) We what an audit entity is, and partially also have a responsibility to the office regarding because the firms do not keep an inventory all questions of conflict of interest, of the work performed by them locally or contract arrangements, acceptance of gifts, nationally. post-employment activities, and so on. We have never gone to court on the The Code applies to all personnel, Code, so we don't how good it would be employees and contract support alike. there. Generally speaking I think we are Abiding by its principles is a condition of satisfied. employment, as well as a condition to every We now have some major issues before contract the office signs. It's part of the us in regard to the Code. For example, training; it's part of the audit manual. recently the office has decided to take the It stipulates that employees are government to court. We've done so required to sign statements that neither because we feel that we have no other they nor their families have interests that choice. We are asked to audit whether the Jacques Goyar 59 Professional and Ethical Concerns in Legislative Program Evaluation value of money h8s been obtained, and the you feel that you have to take the government h8s denied us access to a major government to court. You need a lot of acquisition that was made several years ago "guts," and you've got to make sure you -- a Canadian oil company -- for which know what you are doing. I think that Parliament approved a very specific amount integrity is not about making friends; it's of money. We are suspicious that they not about taking the easy way. Integrity is overspent, and that they didn't have any about doing the job you are asked to do, regard for economy. Prices are reported to even though sometimes it's lonely at the top. be very high cOmpared to what the market would normally pay. I was talking about integrity in the beginning. I think that's integrity: when

Jacques Goyer 60 Professional and Ethical Concerns in Legislative Program Evaluation

There is a famous recipe for rabbit stew that begins with the instruction, "First, catch the rabbit." Too often in reflecting upon ethical behavior in the public service and trying to develop appropriate behavioral standards and codes, we have assumed the Dr. Leigh E. presence of the rabbit, only to discover that we did not have it. And, thus we found that Grosenick we had to settle for watery soup rather than tasty stew. The "rabbit" in this strained analogy is an understanding of the complexity and Doctorate of relativity of ethics within the public arena. Public Administration Program In building the rabbit trap, which is even prior to catching the rabbit, the first myth Virginia Commonwealth we must dispose of is the common University assumption that either we "have ethics" or we "do not." As we enter professions, get elected to legislatures, pursue public purposes through career governmental services, accept political appointments, and in other ways become involved in professional citizenship, we encounter not one but many new universes of behaviors that our upbringing did not prepare us for. We "learn" what is proper professional behavior; we "learn" the etiquettes and conventions of our peers and colleagues. In some cases we learn professional ethics so well that we develop a trained incapacity to view behavior in any way except what we find in our discipline or calling. This "learning" really has unfortunate consequences for the public service, for it tends to result in the enactment of minimum standards of official behavior that can accommodate all of the activities of governmental actors, or it results in impossible general standards of ethical behavior that admit to no guidelines and examples that could be discretely identified. In the former case we are stuck with lists of prohibited behaviors; in the latter with the ethical standards of angels.

61 Professional and Ethical Concerns in Legislative Program Evaluation

Defining the Problem ought to be reached, regardless of what the The problem is to fit the standards to data and analysis suggest. In some other professional behavior in such a way that cases, as evaluation moves to they reflect the values of the nation as implementation, processes and procedures these have been established in 0UI' Consti­ that violate the professional's standards of tution and laws, and, more importantly, to conduct are imposed without due be able to relate professional activities to consideration of what is workable, leading them in such a way that everyday to evasion on the part of the implementors. professional applications are relevant to the In cases of these kinds, and they are all too standards that are set. frequent, neither public nor professional During the past seven years I have values are served. The result is pool' been engaged in systematic research into governance. the behavior of public officials and the Another common problem arises when relationship of this behavior to laws, rules, evaluators and accountants fail to regulations, and professional standards of understand the general rules of conduct that conduct. From this investigation I can govern all public employees, and which are isolate at least five major ethical the special province of the general environments that contain, in some of these administrators. Usually these standards general classes, hundreds of sub-groups of relate to various conflicts of interest that ethically-different behaviors. They are: are not immediate to technical work. An (1) legislators, (2) elected executives and example is the so-called "revolving door." politically-appointed officials, (3) profes­ Here two examples should suffice. Within sionally-educated public administrators, (4) the administrative branch of government, other professionals who are in public agencies regularly entice budget careers, and (5) governmental operatives. professionals away from budget divisions Because of the roles each play, the because of their knowledge of the budget obligations and ethical constraints differ. process and the experience that budget Because of these differences and the professionals bring to certain necessary interaction of each in the subject-matter fields. Budget divisions will governance process there is a tremendous also hire subject-matter specialists away amount of ethical relativity within a system from agencies. There is no conflict of of government in which there are 16 million interest in this activity. It is done public employees, 81,000 units of frequently and for the public can result in government, and almost one-half million better public policy. elected officials. However, an immediate ethical For the professionals who are under problem arises if a legislative program the legislative nmbrella, as most of you are, analyst is offered employment by an agency problems of understanding yOUI' own roles that is undergoing a legislative evaluation, and professional obligations are compounded regardless of whether 01' not the individual by the necessity of understanding every who is offered other employment is other professional obligation. For example, evaluating that agency. Analysts who are in the program evaluator has been that position may have the notion that their professionally socialized not to utilize brilliance has finally been recognized. improper analytical techniques in reaching However, the action itself can be evaluative conclusions. What is propel' 01' interpreted as an employment bribe, improper differs by the nature of the unethical, and totally private-regarding in problem. The accountant should not "cook nature. Also, the administrator who makes the figures," for this is a violation of that kind of offer is in ethical jeopardy. professional honesty. In either case, though Thus, where the practice is common and rarely, there have been suggestions by without penalty in one situation, it is seen elected officials that certain conclusions as sleazy and as a conflict of interest

Dr. Leigh E. Grasenick 62 Professionsl snd Ethicsl Conesrns in Legisletive Progrem Eveluetion

(without even examining individual motives) organization the main elements of a working in another situation. philosophy include references to the: If situations like this were the only problems that we encounter with ethics in • goals and values served by government; the public service, we could deal with them through training, education, and the • relationship of the mission of the enactment of rules. Unfortunately, this agency to the goals and values of example is only one of a vast universe that government; hits us randomly. Since we are beginning to recognize the complexity of the values • values of the ways in which agency environment of government, the next missions are carried out; question is how to deal with it. • values inherent in the technical processes of the agency; The Agency Response A common reaction is for agencies to • relationship between professional develop agency-specific ethical codes of standards and the goals of the agency; conduct, or to fall back upon a professional code and provide more training in that • values that are associated with the code. Another approach has been to ways in which citizens, clients, attempt to take the duties of the agency employees, and relevant others are apart and develop behavioral rules for each treated by the organization; and of the specific problem situations. None of these approaches are very productive as • other factors that are important to incentives to change behaviors, or even to the agency. educate new professionals. They also fall very short of providing the operatives of The advantage of a working philosophy government -- clerks, secretaries and other over a code of ethics or a set of regulations technicians -- with a sense of public and is that it incorporates a system of values agency values. And this is another ignored into the everyday decision-making activities dimension of agency ethics which can be of the agency. Each professional, operative, crucial to both the public's image of the and executive knows that professional agency and the efficiency of internal office activities have a direct relationship to the procedures. goals of the agency, and that the goals of If we are really serious about the the agency have a direct relationship to the incorporation of values and ethics into the values of government. It is this everyday everyday professional environment, we will relevance of values to performance that will begin immediately to develop "working provide a sense of mission and importance philosophies" and training programs that to governmental service. relate professional values to the As an instrument, the working development of public policy and the philosophy is very sensitive. It must be delivery of public services. constantly cared for and adapted to The "working philosophy" has been changing responsibilities, procedural noted as an integral part of the operations advances, technical improvements, and a of those companies who were cited in host of other elements that we tend to treat Waterman and Peter's, In Search of as "mere mechanics," but which in the long Excellence, and even more importantly in run can impact upon public values and the William Ouichi's Theory Z. A working ways in which public values are served by philosophy is a statement that relates the you. values of doing things to the ways in which A working philosophy is also a stern things are done. For a governmental supervisor. If you develop one, advertise it,

Dr. Leigh E. Grosenic/c 63 Professional and Ethical Concerns in Legislative Program Evaluation and then ignore it, you look like a hypocrite "efficient and economical manner" (I hope or worse. One can justify most behaviors everybody is doing that, too). What we need under a general, idealistic code. Working is a reply to the citizen who doesn't care philosophies, however, are very sensitive to how economical and efficient the operations a wide range of professional behaviors. of your agency are. What we need, and The development of a working what a working philosophy can provide, is an philosophy will not take you any longer than honest response to the citizen who says "I the development of an agency code of don't care how economically and efficiently ethics. In governmental agencies it tends to you are providing that service. Why are you be a cooperative effort that can include all doing it at all?" Blaming it on the of the employees, even the operatives, who legislature or your boss will not get you out can develop a sense of mission in of that one. Nor will reference to your understanding that they are participating in code, be it an agency code or a professional an effort that incorporates the ideals of statement of values. Only when you can governance, not merely serving time in relate what you do to the basic value employment. Also, as you begin to nse your statements of government, can you respond working philosophy and relate your actions adequately to that question. A working to it, there will be an easing of the sense of philosophy will enable you to do that. And, frustration that one encounters when when it has been accomplished, you will challenged to jnstify what it is you do. have caught that rabbit. I have long been greatly amnsed by the goals of agencies that relate only to enforcing the law (I hope so, and wish I ~G1l(lli could assume it) and operating in an »)){ M

Dr. Leigh E. Grosenick 64 Professional and Ethical Concerns in Legislative Program Evaluation

The discussion relating to the roles of a program evaluator fits snugly, in this writer's opinion, into the ongoing debate ' about whether public administration is, or should be, a profession and the use of a code Dr. Jack Rabin of ethics by a profession. In this paper, we will concentrate on three questions: (a) Is public administration Graduate Program a profession? (b) If it is, should it in Public Affairs and professionalize? and (c) How can a code of ethics provide control "from within" so that Human Services Administration the profession can police itself rather than Rider College rely on outside regulators?

Is Public Administration a Profession? One major aspect relating to the question of whether ,,, public administration is a ' profession is "What is public administration?" We will not answer this question, as is often done by counterpunching with "If public administration does not exist, then, what are we teaching in MPA and DPA programs?" An answer to the question of whether public administration is a profession is to point to the major sub-fields which constitute the field. These sub-fields (public budgeting and finaiJ.cial management, program evaluation, ' legislative administration, personnel administration and labor relations, city, management, etc.) generally are . securely-defined within most individuaJ:;' minds and are not amorphous as "What is public administration?" Moreover, each sub-field has, or is developing, a well-defined literature (i.e., body of knowledge), and many of the sub-fields have existed for a number of years. Consequently, if one concentrates on tl),e sub-fields of public administration, one is not plagued with the same problems as those endeavoring to define the parent discipline.

65 Professional and Ethical Concerns in Legislative Program· Evaluation

If one can accept that the major superior status....nor is it a sub-fields of public administration are manifestation of a superior well-defined enough, then we will devote "wisdom" in general or of higher our time, when discussing the "field" or the moral character. It is rather "discipline," to the area of program based on the superior "technical evaluation. competence" of the professional If we assert that public administration man. He often exercises his sub-fields can be (or already are) authority over people who are, or professions, we will be asked to state our reputed to be his superiors in criteria (or definition) for determining so. social status, in intellectual Here is where a great deal of the current attainments or in moral literature on professionalism in public character. This is possible administration breaks down, for too often because the area of professional the discussion on whether public authority is limited to a administration, or in our case program particular technically defined evaluation, is a profession centers on sphere. It is only in matters whether the field satisfies a set of criteria touching health that the doctor is developed by sociology. 1 by definition more competent These criteria ordinarily revolve that his lay patient, only in around such items as possession of (a) a body matters touching his academic of knowledge, (b) a code of ethics, (c) speciality that the professor is community sanction, (d) an altruistic nature superior, by virtue of this status, for service, and~ssibly 17 other to his student. Professional characteristics. The preceding four authority. like other elements of criteria are widely used by those favoring the professional pattern, is the sociological, "constellation of characterized by "specificity of characteristics" model. 3. function:' The technical Nevertheless, as Robert W. competence which is one of the Habenstein4 and others have argued, there principal defining characteristics may be at least two other, equally-valid of the professional status and role ways of defining a profession. One approach is always limited to a particular is attributed to Talcott Parsons' and is "field" of knowledge and skil/. known as the "functional model." Consider This specificity is essential to the Parson's functional approach: professional pattern no matter how difficult it may be. in a given There is a very important sense in case, to draw the exact which the professional boundaries of such field. practitioner in our society [Emphasis added.]" exercises authority. We speak of the doctor as issuing "orders" Not having the space to review yet even though we know that the another competing way of defining a only "penalty" for not obeying profession, let us pause and reflect that it is them is possible injury to the possible that program evaluation may be a patient's own health. A lawyer field with "specificity of function," and that generally gives "advice" but if the parenthetically it also contains a growing client knew just as well what to body of knowledge, and community sanction. do it wonld be unnecessary for Can we ignore the characteristics of him to consnlt a lawyer. This altruistic service and code of ethics? professional authority has a Altruism may be part of the "sense of the peculiar sociological structure. It calling" which some practitioners may have is not as such based on a generally as their motivating force in joining the

Dr. Jaclc Rabin 66 Professional and Ethical Concarns in Lagislativa Program Evaluation

public service. A code of ethics is a product be defined by any set of characteristics, the of having a profession, rather than a profession should be able to work with civil precursor, and is the subject of a later service systems. section of this paper. In addition, entry into the profession will not be the only area of If Program Evaluation accommodation. Take performance appraisal. The profession will need to work Is a Profession, with civil service systems to develop Should It Professionalize? procedures whereby tests given the This question concerns testing and profession would be used as part of the control over entering the profession, two appraisal process. common elements taken on as an occupation What is the answer to the question "If professionaliz£'s.. program evaluation is a profession, should it professionalize?" In the long run, the Testing. A profession with a body of answer to this question will depend on (a) a knowledge should not have much trouble consensus among practitioners and with developing tests. Here, one of the academicians that items relating to early contributions to the program professionalization, such as control over evaluation profession could be a glossary entry, code of ethics, and testing, are drawn from the major textbooks and beneficial and (b) political action by those government documents. The glossary could same individuals. However, has the tarnish be submitted to a large panel which would which dulled professionalism's image in the make the final judgments over correct 1960s worn off sufficiently so that the definition. The glossary then could be used professionalism banner is still intact? For as the touchstone for future books, articles, notes relating to this important point, let us and professional reports. Control over proceed to the next section. entrance into the profession is more difficult. Helpful Ideas Controlling Entrance. Most Developed in the Latest occupational groupings which are recognized Professionalism Debate by the public as professions have controls Although many "true professions" governing the number of new entrants into (medicine, law) have been sullied by the field. Indeed, the controls often relate revelations regarding personal income, no to the quality and quantity of knowledge greater criticism was levied than that these which each neophyte must possess. The professions froze out minorities and women program evaluation profession may not be so from their ranks. This equity issue is much fortunate. more important in the public sector, Entry into the public service is especially since notions such as controlled, of course, by civil service "representative bureaucracy" have been systems, regulations, and public laws. Thus, used as the basis for affirmative action in a civil service system may rely on the employment. How can the program profession for continuing education and for evaluation profession deal with this matter? testing, but still retain control over entry The issue can be handled by asserting into the profession itself. In addition, that a profession, with a code of ethics chances are that great changes in civil approved by civil service systems, can be service laws will not take place. the means for ensuring the continuation and How can a program evaluation advancement of affirmative action. How? profession accommodate these realities? By Consider the affirmative action accommodation -- that is, by accepting the experience in government over the last obvious and, since the profession need not decades: laws have been passed, but

Dr. Jaclc Rabin 67 Professional and Ethical Concerns in Legislative Program Evaluation legislatures do not implement the laws; laws governing individual behavior, while courts have issued thousands of orders and legislative committees use their power of have settled numerous disputes, but courts investigation to control behavior. cannot implement their decisions. The Moreover, the legislature, through the situation is even more serious in that power of the purse, has an immense control legislatures can be swayed by different over individual action. Witness the political tides and also that the courts "read control-oriented, line-item budget's the headlines" and weigh public opinion. existence. A profession conld establish a code of Finally, elected executives have ethics, one item of which relates to pressed during the last nine decades for the affirmative action. Through political establishment of OMB-like agencies which action, it may be possible for the profession serve, in part, as watchdogs over individual to convince legislatures that the profession actions. Management information systems be permitted some measure of control over also serve executives' needs for centralizing the behavior of its members, especially with decision-making and, thus, exercising regard to ensuring the representativeness of control. the work force and the validity of These activities are, essentially, occupational tests. "controls from without." They rarely are Since the profession may have great devices initiated or vigorously supported by trouble in dealing with legislatures, another agency people. In addition, most of these action conld be taken by the profession to actions are post hoc in nature -- some benefit affirmative action. This decision decision has been made already by an wonld be to "grandfather" sufficient official or employee, and the control or numbers of minorities and women into the remedy comes afterwards. The controls' early ranks of the profession so as to effect (budget and information systems may guarantee a voice at least in the be exceptions) before or during the profession's policymaking·. Since decision-making process is unclear. grandfathering has been a process used by many nascent professions, the program Controls from Within. Regnlation of evaluation profession conld give some individual action from within may have a thought to it in order to build the ranks of greater chance of affecting the the profession early as well as to settle in decision-making process as it occurs than many minds the problem of equity and those activities being brought to bear from access to the profession. without. Some controls from within exist The Role of a Code already in many government organizations. of Ethics in a Profession For instance, obtaining advisory opinions Historically, government employees from auditors, the use of inspectors general, have had their actions controlled either and the utilization of accounting controls from within or from without the are means of controlling from within. organization. However, none of these devices concentrates on the pressure which Regulation from Without. Today, the individuals can bring on their peers. actions of government employees are being A profession's code of ethics can closely scrutinized and infractions made the provide a set of guidelines for individual and subject of law suits, especially under the collective action which acts as a reference rubric of "personal liability." point by which individual members of the Though personal liability law suits may profession can judge member actions a be something relatively new, regnlation priori as well as while those decisions are from without by other means is not. being made. An example of an existing code Legislatures pass Hatch-type and ethics of ethics in the public sector is the "City

Dr. Jack Rabin 68 Professionel end Ethicel Concerns in Legislstive Progrsm Evslustion

CITY MANAGEMENT CODE OF ETHICS

The purpose of tbe Ioternational City Management Association is to increase tbe proficiency of city managers, county managers, and otber municipal administrators, and to strengtben tbe quality of urban government through professional management. To furtber tbese objectives, certain ethical principles shall govern tbe conduct of every member of tbe Ioternational City Management Association, wbo shall: 1. Be dedicated to tbe concepts of effective and democratic local government by responsible elected officials and believe that professional general management is essential to tbe achievement of tbis objective.

2. Affirm tbe dignity and wortb of tbe services rendered by government and maintain a constructive, creative, and practical attitude toward urban affairs and a deep sense of social responsibility as a trusted public servant.

3. Be dedicated to tbe highest ideals of bonor and integrity in all public and personal relationships in order that tbe member may merit tbe respect and confidence of tbe elected officials, of other officials and employees, and of tbe public.

4. Recognize that tbe chief function of local government at all times is to serve tbe best interests of all of tbe people. 5. Submit policy proposals to elected officials, provide tbem witb facts and advice on matters of policy as a basis for making decisions and setting community goals, and uphold and implement municipal policies adopted by elected officials. 6. Recognize that elected representatives of tbe people are entitled to tbe credit for tbe establishment of municipal policies; responsibility for policy execution rests witb tbe members.

7. Refrain from participation in tbe election of tbe members of tbe employing legislative body, and from all partisan political activities whicb would impair performance as a professional administrator. 8. Make it a duty continually to improve the member's professional ability and to develop tbe competence of associates in tbe use of management techniques.

9. Keep tbe community informed on municipal affairs; encourage communication between tbe citizens and all municipal officers; emphasize friendly and courteous service to tbe public; and seek to improve tbe quality and images of public service.

10. Resist any encroachment on professional responsibilities, believing tbe member should be free to carry out official policies witbout interference, and handle eacb problem witbout discrimination on tbe basis of principle and justice.

11. Handle all matters of personnel on tbe basis of merit so that fairness and impartiality govern a member's decisions pertaining to appointments, pay adjustments, promotion and discipline.

12. Seek no favor; believe that personal aggrandizement or profit secured by confidential information or by misuse of public time is disbonest.

Or. Jeclc Rabin 69 Professional and Ethical Concerns in Legislative Program Evaluation

Management Code of Ethics" (see table). code could be nsed as a means of bringing The author does not have any data, however, further control over behavior. as to implementation or acceptance of the Finally, the program evaluation Code by cities. profession, through its code of ethics, could Nevertheless, given the limits on a ensure the nse of affirmative action public-sector profession listed above, a principles and procedures by its members, profession should be an able partner with nse being determined also by the ability of legislatures, appointed and elected profession members to bring political executives, and merit systems in helping to pressure on legislatures. Since the bring controls on members from within the development, let alone implementation, of a public organization. The dilemma is: Will code of ethics is a long-range goal, the this experiment be tried? profession can assist affirmative action by grandfathering in sufficient numbers of Conclusions minorities and women so that the A case can be made that program profession, from the onset, will be evaluation is a profession. Most professions representative. have some type of ethical code; depending on the latitude given the profession, the (&1\\\\ ))))rili Notes

(1) See, Richerd L. Schou (19761. "Public Administration as a Profassion: Problems and Prospects." Public Administration Review 36(May/June); 253-259. (2) See, Jade. Rabin (1981-1982). "The Profession of Public Administration." The Bureaua"ar 10 (Winterl: 10-12. (3) Robert W. Habanstein (1963). "C,itique of 'Profession' as a Sociological Category." The Sociological Quarterly 4 (Autumn): 292. (4) Ibid ., p. 298. (5) Talcou Parsons (1939). "The Professions and Social Structura." Social Forcas 17 (May): 457-467. (61 Ibid .. p. 460. (7) J. David Woodard (1984). "Ethics and the City Manager." The Bureaucrat 13 (Spting): 54.

Dr. Jack Rabin 70 Design and Methodology: How Do Organizations Do It?

Our office began in 1978, but we did not hire a methodologist until 1984. 1 had long wondered about hiring a methodologist. 1 had been aware of the Virginia model for several years, but there were some experiences which led me to decide it was a necessary move. William Thomson The first experience occurred several years ago when we did an evaluation of the Treatment Alternatives to Street Crime Director (TASC) program. The study involved an Performance Audit Division outcome-type measure. We worked through a lengthy process with TASC to identify and Office of the Auditor General define a control group and felt we had a Arizona well-designed study. When the data was analyzed we found the program did not work. At that point TASC charged that the control group, which had previously been jointly agreed upon, was inadequate -- that there were differences between the populations, etc. They launched a major battle against our methodology and sought aid from all quarters. They had people from the universities and the Department of Corrections, which was funding TASC, all attacking our methodology. Worse, we did not have a sound answer or basis for rebuttal. A few years later we were assigned the responsibility of evaluating our vehicle emissions inspection program. Again, 1 felt an outcome measure would be required. (I did not think the legislature was interested in how fast we could process cars or how cheaply we could inspect them, but whether the air was getting cleaner.) In this case we were looking ahead for the methodology we might use on this audit. We sent staff to a course at Arizona State University (ASU) on time series analysis and became acquainted with the professor who taught it. We eventually hired the professor as a consultant to conduct our study. He is not only an expert on time series analysis, but more importantly, he is an expert program evaluator. His expertise was later to prove more critical than we ever expected. We began the evaluation expecting that the data would show a 10-12% reduction in carbon monoxide levels because

71 Design end Methodology: How Do Orgenizetions Do It? of the presence of the program. Some use DMTAGs, we can use a methodologist. national EPA studies had shown similar Second, we were investigating results and we felt Arizona would probably conducting some statistical training for our fall in that range. Imagine our surprise staff. As we looked at that course, we when the consultant called, saying he had realized that you can't adequately train the run the models and could not find an effect. entire staff in statistics because they don't We had originally thought we would run use it often or deep enough to retain the 10-15 models to establish the range of training. lnstead, you need a person whose carbon monoxide reductions. The consultant job it is to be knowledgeable in statistics. ultimately ran several hundred models and We hired a methodologist. never found an effect from having the As I mentioned yesterday, we have 24 vehicle emissions inspection program. professional FTEs and of that, we have Further, he did some reruns of data used in converted half an FTE into a metho­ an EPA study and found the results they dologist. We originally started with a stated were not statistically significant. half-time methodologist for two reasons: Our report's conclusion that the (1) she was finishing her dissertation and did program was not working launched a major not want to work full-time, and (2) we were battle -- both political and methodological not sure if we would have enough of a -- with the Department of Health Services, workload for a full-time position. which ran the program, with some Although she has now finished her legislators, and even with the press. (One doctorate, we are finding substantial editorial, in fact, attacked us for our results benefits from continuing the position as a and suggested we should be investigated by part-time position. We find the half-time a grand jury.) This time, however, our position lets her participate in activities methodology stood the test. 10 fact, the that continue to maintain and develop her Department of Health Services contacted skills. She teaches part-time, and has also ASU asking for professorS to'rebut our recently been awarded a federal grant to do study, but could find none. This cinched for a program evaluation of drunk driving me, once and for all, the importance and programs. We also find, as I will discuss critical nature of advanced methodology later, that maintaining her academic when we needed it. contacts brings important benefits to our office. 10 fact, if we develop a full-time workload for our methodologist, we would Recognizing the Need seriously consider simply adding another for a Methodologist half-time person. After the vehicle emissions audit we The methodologist functions at the began to consider whether we needed an senior or team leader level. We placed our in-house methodologist or whether we methodologist in this level for two reasons: should hire outside consultants as needed. One, we felt we needed that salary range to The move to hiring a methodologist was provide an adequate salary. The salary finally sparked by two things: I saw that the range for senior auditor begins at $29,000 General Accounting Office was starting and goes to $39,000. Two, we felt she design methodology and technical assistance needed the equal status so she could work groups (DMTAGs). I had been aware of with team leaders. JLARC's use of a methodologist but felt We have been fortunate to obtain a JLARC filled a different role -- more of a very, very qualified methodologist. She has research role -- than we did. Therefore, I a doctorate of public administration, but was not sure how much we would use a more importantly, her emphasis has been methodologist in our office. However, GAO almost entirely in program evaluation. She mostly does the same "meat and potatoes considers herself a social science researcher work" such as we do. I figured if GAO can and evaluator. She has taken probably every

William Thomson 72 Design end Methodology: How Do Organizations Do It?

class of advanced statistical program We are also using Barbara to design evaluation at ASU and was considered their some studies. We have a statutory top graduate student. requirement to evaluate our mandatory motor vehicle insurance law. Among the The Methodologist's Role things we must examine are the impact of When we brought our methodologist in, that law on the number of uninsured we used an informal method of integrating motorists, insurance rates, and court costs. her into our system. We have not formally She is now researching the types of data required that her services be used; rather, currently available and whether additional we have integrated her use as we did our data will be needed, and she is beginning to computers -- we have made her available design the methodology we will use. and encouraged use. We have, however, asked certain questions that almost require the teams to Essential Attributes consult with the methodologist. We have of a Methodologist also tried to encourage the teams to use her if you are thinking of hiring a by not charging her hours against the budget methodologist, there are some essential hours for their projects. Her services can attributes to consider. Perhaps the most be used to as large an extent as needed important is people skills. I know that without counting against the budget. sounds funny, but you must ensure that the Part of the reason for not mandating methodologist can work with staff. Barbara use was we felt there was a learning curve is one of the best fits we could have had for for both sides. Barbara, our methodologist, our organization. The staff love her and it needed to learn what auditing was in our works out well. When I talked to Bill sense. We needed to learn methodology Johnson from GAO, he told me to be very from her. careful to get people used to working with Barbara functions foremost as an staff. He said you can't afford to have the in-house consultant, particularly on statistician mentality, the person who is the methods. She reviews our sampling, our technical expert, looking down on the staff. statistical analyses, and where necessary, That's really true. she sets up the research design for us. She Methodologists also need some very has also done a lot of analysis of data. Staff good program evaluation skills. I think you take her on preliminary surveys to review are wrong to think of a methodologist as a data, particularly data on the computer, and statistician. Think in terms of a social request her to review the types of analyses science researcher when you look for a that are appropriate given the data methodologist. Someone who is used to available. Barbara also trains staff in the research design, interested in program use of SPSS programs on our pes, and evaluation, who can work with fuzzier data research design. than maybe pure statistics. Of course, they She has become an important element do need good statistical skills. They need of quality review. She has been helpful in the state of arts statistical skills. They recent reports in determining what we could need to know their time series analysis and say -- the limits of the language we could other high-level techniques. use, the limits of the conclusions we could draw -- given our data. She also serves a The Benefits quality review role in our dealings with our There are a lot of benefits to having a consultants, particularly in evaluating the methodologist. Obviously, one is work methods they use. She not only helps us quality. The accuracy, reliability, and draft RFPs and evaluates proposals, but soundness of our data, and the soundness of serves as a quality control check on our our analysis have been improved. It has consultants. allowed us to do new types of work we

William Thomson 73 De.ign elid Methodology: How Do Orgenizetions Do It?

simply did not have the expertise to do applicants. A number of highly qualified before, such as audits addressing outcome people have come to us applying for jobs. measures. (I think outcome measures are We've probably hired three or four of them the most faScinating, the most crucial, and in the last year. Many of them were out of the most difficult to do. You need a a different track than we often considered. methodologist to do them.) They weren't the usual MPA or MBA Another benefit has been to increase student, they were persons working on staff confidence. They appreciate having doctorates, persons interested in research someone with Iter expertise to back them and evaluation. They became interested in up. She has also given us increased us because they knew Barbara and heard credibility tl> our auditees and to outside about our work. parties. Her presence on our staff has Barbara has also been a conduit for really enhanced and developed our rapport other resources. She has great contacts with the unlversity. They recognize her with faculty who are now contacting us, expertise and see our interest in increasing asking if they can work with us to do some our sophistication. of their public service hours. An interesting side effect of that is, because of her contacts and the respect she ~& has earned, she has become a conduit for job }\((( mdi§

William Thomson 74 Design and Methodology: How Do Organizations Do It?

This panel addresses an important issue for program evaluation and post auditors. Specifically, how can the quality of our evaluation work be improved and what are the ways in which various legislative program evaluation agencies integrate research methodology into the audit process? The need for improved research in Bernie Geizer dealing with complex public programs goes far beyond those involved solely in evaluation research. As programs became Director more complex and required increasing Legislative Commission financial resources to carry them out, public administrators sought to use soundly on Expenditure Review designed and executed studies that would New York State Legislature increase the likelihood that their analyses were on target. New York's Legislative Commission on Expenditure Review (LCER) was one of the first legislative program evaluation agencies to recognize the need for integrating sound research methods into its program eval- uation work. For example, to the best of my knowledge, LCER was the first legislative program evaluation agency to apply systematic user survey techniques to measure the impact of public programs. Over the years, LCER has used case studies, survey research, content analysis, sampling, comparative analysis, time series, and statistical analysis besides the more mundane file reviews to measure the effectiveness of public programs. The "Diffused" Approach The three organizations participating on this panel all recognize the importance of using sound research methods in their studies, but have organized quite differently to accommodate the input of improved methods in their research. Our methodological input at LCER has been

7S Design end Methodology: How Do Orgenizetions Do It? characterized as "diffused" throughout the members can therefore serve as participants organization, Le., there is no single in research design development instead of methodologist or methodological unit which merely as "after-the-fact" reviewers of reviews all of the work being carried out by research strategies. We expect the audit LCER researchers. team and the audit review committee to Our research methods review is "hammer out" the research methodology carried out in two ways: (1) by input to the which will produce the desired results while program audit team by another staff being mindful of audit report deadlines and member of LCER on an informal, in-house the availability of staff resources. consultant basis, or (2) by having LCER's Since all of the members of the audit audit review committee use a resource review committee have had "hands on" person on staff with a background in applied experience with their own audit work -­ social science research. One of the reasons often in the same area being examined by we feel comfortable with this approach is the audit team -- we expect that they will that we have some members of staff -­ be able to assist the audit team in making including several PhDs -- who have had a assessments about the quality of data and substantial amount of course work and other problems that are likely to be experience with social science research encountered in doing the study. We will also methods and who have demonstrated the suggest -- and sometimes direct -- these ability to design high quality evaluative audit teams to meet with other LCER staff studies and to make constructive members who serve as "in-house" suggestions concerning research methods. consultants for specific needs such as The audit review committee is an reviewing questionnaires, for sampling integral part of the process in carrying out strategies, or for adapting audit work to our our audit work. The function of this microcomputers. committee is to provide guidance to audit Finally, after the research design has team members from the inception of the been developed, the audit team is not cast audit work through completion of the audit adrift, because we have monthly meetings in report. This committee -- which is which the audit team advises the audit composed of the Director, the Deputy review committee of its problems and Director, a senior member of the LCER progress. These meetings provide an staff and one other person -- makes opportunity to make modifications to the constructive suggestions to the audit team audit research and deal with new issues as concerning the research design and program they arise. and policy issues that may arise during an The audit review committee also is audit. Except for the Director and Deputy responsible for reviewing the first draft of Director -- who sit on all audit review an audit report to see that research methods committees -- the composition of the audit have been adequately and appropriately review committee changes as do the audit applied and critically examines policy issues teams. Our intent is to staff audit teams and audit findings and recommend ltiOns. and audit review committees with Additionally, working papers are reviewed researchers who are strong in research by LCER's Deputy Director to assure the methodology and who have had some adequacy of documentation for audit experience or exposure to the program being findings. audited. Conceivably, a "reviewer" for one audit may be an audit team member for Attributes of the Professional Staff another audit; this does not detract from Since there is no methodologist, per the frankness of the working sessions. se, on the LCER staff, we have not sought It is important to point out that the out individuals whose academic training or audit review committee is created before work experience is largely in research the research design is developed, and its methods. Instead, because of the diffused

Bernie Gsizer 76 ------_. Design lind Methodology: How Do Organizations Do It? nature of our methodological expertise, we methodologist "has no sense of field work," seek to employ people who have the skills that he or she "doesn't have a feel for soft required to be good, generalist program data," and that the methodologist "is not evaluators and researchers. concerned with audit scheduling and The attributes we look for in our deadlines." professional staff are: good judgment, Second, there is a greater appreciation intelligence, inquiring minds, tact, and good developed among staff members related to methodological, quantitative and writing using different research strategies and skills. With respect to quantitative skills, methods in audits and program reviews. we are finding that virtually all new staff Finally, the acquisition of improved members with a graduate degree in public research skills by staff members not only administration or management have greater makes them more valuable members of staff quantitative skills than the graduates of but also promotes their own professional similar programs of only a few years ago. In development and their marketability. LCER my opinion this also has contributed to an is committed to training to assure that overall improvement in the quality of our those skills previously learned are research designs and our audit reports. appropriately applied to program auditing. Based on our experience, we believe The disadvantage of this diffused that staff members with "hands on" approach to audit research is that we may experience in doing audits and dealing with sometimes need to go outside our agency data and information systems can organization and hire a research consultant develop a realistic assessment of what for his expertise in carrying out a very research methodologies can be used to give specialized research project. We have done us a quality product, meet the deadline this on several occasions and we have been established by the Commission, and carry pleased with their work. We expect that out the work with the staff resources this situation will occur from time to time available to the Commission. These two in the future, and we will again seek outside latter points -- getting the audits done on consulting services as the need arises. time with available staff resources -- are We believe that the organizational key factors in scoping our studies and in structure of LCER.JWd its long standing selecting the research strategies to be used. commitment to high quality research -­ within the very real constraints of project Advantages and Disadvantages deadlines and the availability of staff We believe there are some important resources -_. has enabled us to produce benefits resulting from this approach. First, accurate audit reports and objectively it is our opinion that tension is reduced present the information that the New York nsing our approach compared to the tension State Legislature needs for its oversight that often exists between the members of responsibilities. an audit team and a designated method­ ologist. Using our approach, audit staff are ------~ not likely to complain that the staff

----_._------Bernie Geizer 77 Design and Methodology: How Do Organizations Do It?

The task of generating the design and methods for studies undertaken by the Joint Legislative Audit and Review Commission (JLARC) has changed since the organ­ ization's inception 12 years ago. While the locus of the activity and responsibility lies with the project team, additional resources are brought to bear during the research Dr. Gary Henry process. The shift from a decentralized, team-oriented process to one that is centralized and with more organizational Joint Legislative inputs has occurred through accretion. However, three distinct stages can be noted. Audit and Review Commission During the first stage, the team Virginia worked to pull together the project scope and the research workplan without outside assistance. The Director and, in time, other managers reviewed the team products mainly for issue orientation. Design and methods were left to team members. This served rather well for straightforward projects, but the field was becoming more methodologically sophis­ ticated. Higher standards for information, and techniques that could provide more direct and conclusive answers to legislative questions were beginning to surface. These couditions led to the initiation of a second stage in which a method­ ologist was hired to work with teams in developing the methods for a study. Typical questions for the methodologist related to the construction of questionnaires, the selection of efficient sample sizes, and analysis strategies and techniques for trend analysis or testing differences in means or variance. The methodologist was called in by the team, or when a manager saw the need for some consultation. Most of the methodologist's time was spent on his assignment as a team member for a project. The methodologist's role at this stage can be described as a consultant or technician.

78 Design snd Methodology: How Do Orgsnizstions Do It7

This was valuable -- when the team • ensure the validity of the study elected to use the information -- for findings, improving the quality of information, but did not pull the methodologist into the • ensure methods yield findings on conceptualization of the design. Therefore, issues, not just information, opportunities for more powerful designs depended solely on the team. • help project teams focus on The third stage of development sought salient, researchable issues, to bring methodological resources outside the team into the conceptualization and • provide training and support for design process for all studies, as well as to more sophisticated analysis, continue the technical advice. During this stage the methodologist role took on • where required for projects, carry full-time responsibilities. Eventually, a out sophisticated, quantitative computer operations analyst and another analysis, full-time methodologist were added to increase the methodological capacity of the • review documents for appropriate organization. Centralizing the methods use of results, and functions in this way required a definition of mission and purpose for the section and gave • make sure design, data, findings, it a unique role within the organization. and conclusions are defensible.

These objectives integrally link the section Mission and Purpose with the teams. Both groups bear respon­ Establishing the Methods Section sibility for the product. Yet each has a within the organization could work only if different role that is useful in reaching the the section contributed something to the organizational goals. product for which it could be held accountable. Also, it could not duplicate team or management functions. The Methods Section was given the general The Methods Section's Role responsibility for the "methodological in the Organization quality" of each study. This responsibility It became obvious with the inception made it necessary for the teams and the of a specialized methods function within section to work together in designing and JLARC that the role of the section would be implementing to achieve common goals of a flexible one. Even if three methods relevancy, quality of information, practi­ specialists could keep up with six or seven cality, and timeliness. teams and the 20 or so analysts on a The general statement of day-to-day basis, it would not be desirable. responsibility and the common goals guided The section was not established to manage the original development and subsequent or duplicate existing functions. Therefore, refinement of the section's mission. Eight the selection of process points to be objectives currently guide the section's involved with the teams and the extent of work with the research teams: involvement were crucial. Three factors have guided the level of activity the methods specialists have had • provide the most conclusive with teams: (1) important study issues answers to the legislature's where more sophisticated methods could be questions with the least used, (2) staff assigned to the team, (3) and complicated methods, organizational priorities.

Dr. Gsry Henry 79 Design and Methodology: How Do Organizations Do It?

Kinds of Issues organizational priority for a particular Issues in JLARC studies can be study. Almost all studies could have categorized in three ways. Often issues fall elements requiring sophisticated analysis. in the category of program reviews. Usually Whether the study garners the resources program reviews are conducted in a manner from the section to facilitate this analysis that makes necessary qualitative judge­ depends upon the organizational priority for ments about compliance with standards or the study. When legislative interest is high legislative intent. The technician's role for for impact analysis, the methods section surveys and analysis is usually the extent of dedicates significant time and energy to the involvement. Usually, these issues addressing the issues. receive less attention from methods specialists, although productivity and comparative staffing issues are exceptions. The Potential for Conflict Evaluation research issues focus on While the team role and the section program impacts or outcomes toward role within the organization are different, achieving objectives. These issues require sometimes tensions arise from interactions. an adherence to concerns for causality, Both team members and specialists because the crux of such an issue is the recognize the importance of quality impact of the program on certain research and timeliness. However, the measurable objectives. During concep­ emphasis on how to make the trade-off is tualization and design of these study issues, different. Specialists are trained to address methods specialists work closely with the validity, reliability, and defensibility of the team, bringing their particular expertise in research first and foremost. Teams and sorting out cause-effect relationships. management have a keen awareness of The third type of issue involves policy deadlines. When the priorities placed on analysis, or a future-oriented look at the these values come into conflict, they are consequences of policy alternatives. These sometimes difficult to reconcile. issues tend to be quantitative and rely on Three factors within the organization construction of models to represent some generally bring about a fair resolution to the part of a policy's impact. These studies conflict. Both the division chief (the teams' receive more of the specialists' time manager) and the chief methodologist report throughout the project. directly to the Director. This gets the highest level of management involved in the Assignment of Staff resolution and ensures both sides of equal The second factor that influences the footing in the decision. Secondly, both sides role of the methods specialists is the staff in the conflict recognize and respect the assigned to the project. The technical side pressure on the other (usually). The conflict of the role increases for teams that are less has been discussed thoroughly, and both strong in quantitative methods, but the sides are more interested in a decision that other work is less, except of course when a is in the organization's best interest than in sophisticated part of the study is assigned to winning. the specialist. When there is a high Finally, both the team and the potential for the study team to conduct methods section know the resolution must quantitative analysis, specialists from the come from the organization. The decisions methods section usually work closely with usually have implications for the organi­ the team to ensure that the techniques are zation, and therefore the resolution applied appropriately. appropriately belongs at that level. Furthermore, organizational decisions tend Organizational Priorities to involve some compromise. Standards The third factor in determining the that are likely to affect validity, credibility, role of the methods section is the and defensibility are upheld. Resources are

Dr. Gary Henry 80 Design and Methodology: How Do Organizations Do It? sometimes added to aid the team in meeting Conclusion the standards. Standards that are not likely The transition from a decentralized to affect the study at hand are relaxed. process for research design to a more centralized process has occurred smoothly. Teams frequently request more time than the methods specialists can provide. More Lessons Learned methodologically sophisticated projects Through the process of establishing an have attracted legislative interest. organizational role and identity for the Legislators seem to recognize our ability to methods section, a number of lessons have produce studies that provide a sound basis been learned. The first lesson is that you for decisionmaking. Therefore, teams seek can't be arbitrary. Reason and logic to address larger study questions, when dominate the discussions. You can't expect relevent, and involve the methods section to win all the time. Both sides in the personnel to do so. conflict are trying to do a competent, The presence in legislative debates of professional job. Neither is always right. high-quality information from JLARC You can't stay angry. In a small research has increased the level of organization, setting up barriers after losing expectation for information from the in a conflict can be devastating. We must agenices. Executive branch agencies are come back the next day and work together improving data bases to provide information for the job to be completed, so anger seems that is more reliable and relevant. They are to dissipate as we focus on the project. developing more rigorous analytical Perhaps the most important lesson techniques. Unquestionably, the Com­ that has been learned is that in the end the mission's interest and support of process and the resolution have been information of a high methodological caliber reasonable. Products seem stronger, and has led to improvements beyond the work of the work environment is oriented to growth the JLARC staff. and drawing a wide range of relevant skills from the staff. CG} $\\\ Ji1f1$j

Dr. Gary Henry 81 82 Appendix A

1986 SESSION LDll48324

SENATE BILL NO. 173 Offered January 20, 1986 A BILL to amend and reenact §§ 30-fi6. 30-fi7. 30-70 and 30-71 of the Code of Virginia and to repeal § 30-73 of the Code of Virginia. the amended and repealed sections relating to the Legis/ative Program Review and Evaluation Act.

Patrons-Willey, Andrews, H. B., Buchanan, and Babalas; Delegates: Morrison, Ball, callahan, Wilson, Putney, QUillen, and Smith

Referred to Committee on Rules

Be it enacted by the General Assembly of Virginia: 1. That §§ 30-66, 30-67, 30-70 and 30-71 of the Code of Virginia are amended and reenacted as follows: § 30-66. Functional areas; sCheduling of study areas.-A. The functional areas of state government shall be scheduled for legislative review and evaluation by the Joint Legislative Audit and Review Commission as specified in paragrapll subsection B, on a seven-year cycle, and beginning in the 1979-80 fiscal year. B. Beginning with tlIe l-9+ll legislative sessian, aIHl From time to time as may be required, the Senate and House of Delegates shall by joint resolution establish a schedule for the review of the functional areas of state government. In the absence of a resolution, the Joint Legislative Audit and Review Commission shall select a functional area for review on an annual basis. § 30-67. Discretionary selection procedure; coordination with standing committees; expenses.-A. l'xeepl fM tlIe pi-Iet review pravillell fM in tIHs coallier, aIHl Prior to the year in which a functional area of government is designated to be scheduled for review, the Joint Legislative Audit and Review Commission shaH e&HSe to Be intralllleell may provide for the introduction of a joint resolution which shall identify to the extent feasible the agencies, programs or activities selected for review and evaluation from the functional area. B. To ensure coordination of the review and evaluation activity with appropriate committees, the resolution specified in paragrapll subsection A shaH may identify each House and Senate standing committee to be invited to participate with the Commission in designing such studies as will be carried out from the scheduled functional areas. e. The compensation and expenses of the members of cooperating committees or sUbcommittees necessary to accomplish the functions specified in paragrapll subsection B shall be paid from funds appropriated to the Commission. § 30-70. Reporting; hearings.-A. The Joint Legislative Audit and Review Commission shall pUblish and submit its reports with appropriate findings and recommendations to the Governor and members of the General Assembly, and shall transmit them to the House and Senate standing committees identified by resolution in § 30-67. B. The standing committees shaH may hold a public hearing on reports prepared pursuant to this chapter witllin ~ Gays at their earliest convenience after the date of transmittal. Hearings may be held jointly or singly by the committees.

83 Senate Bill No. 173 2

C. The standing committees shall hear testimony from the Commission, agency and program representatives, the public in general, and such others as may be deemed appropriate. § 30-71. Hearing criteria.-At each hearing reEillireEl By which may be held pursuant to § 30-70, the standing committee conducting such hearing and the agencies testifying shall respond to, but not be limited to consideration of, the following questions: 1. What are the problems, needs, or missions that the program is intended to address and what has been accomplished? 2. What is the effect of the program on the economy including but not limited to: competition, unemployment, economic stability, attraction of new business, productivity, and price inflation to consumers? 3. Would the absence of any regulatory activity significantly harm or endanger the pUblic health, safety, or welfare? 4. Has the program or agency carried out its mission in an efficient, economic, and effective manner? 5. What services could be provided and what level of performance could be achieved if the program were funded at a level less than the existing level? 6. What other state programs have similar, duplicate, or conflicting objectives? 7. What federal activities have similar, duplicate, or conflicting objectives? 8. How does the agency ensure that it responds promptly and effectively to complaints concerning persons affected by the agency? 9. To what extent have the agency's operations been impeded by existing statutes, procedures, or practices of the Commonwealth of Virginia, or of other state agencies? 10. What action plans have been or are being proposed to improve agency operations where the need for improvements has been identified in previous executive or legislative oversight studies and reports? 2. That § 30-73 of the Code of Virginia is repealed.

Official Use By Clerks Passed By Passed By The Senate The House of Delegates without amendment 0 without amendment 0 with amendment 0 with amendment 0 sUbstitute 0 sUbstitute 0 sUbstitute w /amdt 0 sUbstitute w/ amdt 0

Date: _ Date: 1

Clerk of the Senate Clerk of the House of Delegates

84 (G}l[Q»)[1j)) (Gi1([Q»)[~) CG)?1([Q»Jric)) LEGISLATIVE PROGRAM EVALUATION SECTION REGIONAL MEETING Tuesday, October 15, 1985 8,00 - 9,00 a.m. Caffee (John Marshall Room) CONFERENCE 9,00 - 9,15 a.m. Welcoming Remarks, Ray D. Pethtel, Director, JLARC 9,15 - IO.30 a.m. PROFESSIONAL AND ETHICAL CONCERNS IN ON LEGISLATIVE PROGRAM EVALUATION LEGISLATIVE Moderator,Ray D. Pethtel, Director, JLARC Panelists, Jacques Goyer, Principal, Office of the Auditor General of Canada OVERSIGHT Dr. Leigh Grasenick, Virginia Commonwealth University Dr. Jack Rabin, Rider College

IO.30 - 12,15 p.m. DESIGN AND METHODOLOGY - HOW DO ORGANIZATIONS DO IT! The ):> -c Joint Legislative -c 00 Moderator, Philip Leone, Deputy Director, JLARC CD en :::J Panelists: William Thomson, Director, Performance Audit Division, Audit and Review a Office of the Auditor General, Arizona X Commission 0: Bernie Geizer, Director, Legislative Commission of the on Expenditure Review, New York Virginia Dr. Gary Henry, Chief Methodologist, JLARC General Assembly 12,15 . 1.30 p.m. Lunch (On Your Own) 1,30 - 4,15 p.m. CONCURRENT WORKSHOPS (General Assembly Building) (I) Evaluating Secondary Data Hotel John Marshall (Dr. Debra Rag and Jay Landis, JLARC) 11th Floor Conference Room October 13 . 15, 1985 (2) Graphic Design for Information Presentation (John Long and Dave Porter, JLARC) 4 West Conference Room (4th Floor) (3) Survey Design (Dr. Gary Henry, JLARC) 10th Floor Training Room 4,30 - 5,00 p.m. Tour af Virginia State Capitol Building

( ------, ( ----J ~ ------) ( J ( ----J (--- J Sunday, October 13, 1985 12,15 . 1,30 p.m. LUNCH (Roof Garden) 4,00 . 5,30 p.m. Registration (John Marshall Room) Introductory Remarks, Senator Edward E. Willey, 6,00 . 7,00 p.m. Reception Vice-Chairman, JLARC 7.15 . 9,00 p.m. Dinner in Honor of Conference Co-Hosts, Delegate A. L. Philpott, Speaker, "Oversight in the Congress and the U.S. General Speaker of the House of Delegates Accounting Office," Eleanor Chelimsky, Director, Senator Edward E. Willey, Program Evaluation and Methodology Division, President Pro Tern of the Senate, U.S. General Accounting Office Delegate L. Cleaves Manning, Chairman, JLARC 1,45 . 3,30 p.m. INGREDIENTS FOR Speakers, "Status of Legislative Oversight in the States" AN EFFECTIVE OVERSIGHT FUNCTION Dr. Alan Rosenthal, Eagleton Institute of Politics, Rutgers University Moderator, Dr. Alan Rosenthal, Eagleton Institute of Politics

ll IIStatus of Sunset in the States j Mr. Richard Jones, NCSL Panelists, Dale Cattanach, State Auditor, Wisconsin George L. Schroeder, Executive Director g; Monday, October 14, 1985 South Carolina Legislative Audit Council William Thomson, Director, 9,00 . 9.45 a.m. Registration and Coffee (John Marshall Room) Performance Audit DivisIOn, 9,45 . 10,00 a.m. Welcoming Remarks Office of the Auditor General, Arizona Delegate L. Cleaves Manning, Chairman, JLARC John W. Turcotte, Director, Joint Legislative Committee on Performance 10,00 . Noon NEED FOR LEGISLA TIVE OVERSIGHT Evaluation and Expenditure Review, Mississipoi and LEGISLA TOR EXPECTA TIONS 3,30 . 3,45 p.m. Break OF EVALUATION STAFF 3,45 . 5,00 p.m. DIRECTIONS FOR OVERSIGHT IN VIRGINIA Moderator, Delegate L. Cleaves Manning, Speakers, L. Cleaves Manning, Chairman, JLARC Chairman, JLARC Ray D. Pethtel, Director, JLARC Panelists, Representative Wayne Fawbush, Oregon Commenrs, Senator Edward E. Willey Senator Mordecai Lee, Wisconsin Representative Herbert F. Morgan, Florida Senator Hunter B. Andrews Delegate Richard M. Bagley 5,30 . 7,00 p.m. Reception (The Downtown Club) ( j ( j ( j JLARC STAFF

RESEARCH STAFF ADMINIS1RATIVE STAFF

Director Section Manager Philip A. Leone Joan M. Irby, Business Management & Office Services Deputy Director Kirk Jonas Administrative Services Division Chief Maryann Craven Glen S. Tittermary Secretarial Services Section Managers Bonnie A. Blick Gary T. Henry, Research Methods Rosemary B. Creekmur & Data Processing Betsy M. Jackson .John W. Long, Publications & Graphics VelmaM.Lee

Project Team Leaders SUPPORT STAFF Stephen W. Harms Clarence L. Jackson Barbara A. Newlin Technical Services Robert B. Rotz R. Jay Landis, Data Processing • David W. Porter, Graphics

Project Team Intern William A. Butcher Karen Widener Karen E. Chappell Nolani Courtney Stephen Fox Lynn L. Grebenstein Thomas J. Kusiak Susan E. Massart Gregory J. Rest Cynthia Robinson Carl W. Schmidt E. Kim Snead • Indicates staffwith primary Geraldine A. Turner assignment to this project RECENT REPORTS ISSUED BY THE JOINT LEGISLATIVE AUDIT AND REVIEW COMMISSION

Managemenr and Use of Consulranes by Scare Agencies, May 1980 The General RcJief Program in Virginia, September 1980 Federal Funds in Virginia, October 1980 Federal Funds, A Summary, January 1981 Merhodology for a Vehicle Cosr Responsibiliry Srudy, An lnrerim Reporr, January 1981 OrganiZJlrion and Adminisrrarion of rhe Deparrmenr of Highways and Transporrarion, An lnrerim Reporr, January 1981 Tiele XX in Virginia, January 1981 OrganiZJlrion and Adminiserarion of Social Services in Virginia, April 1981 1981 Reporr to rhe General Assembly Highway .md Transporrarion Programs in Virginia, A Summary Reporr, November 1981 Organizarion and Adminisrrarion of rhe Deparrmene of Highways and Transporrarion, November 1981 Highway Consrrucrion, Mainrenance, and Transir Needs in Virginia, November 1981 Vehicle Cosr Responsibility in Virginia, November 1981 Highway Financing in Virginia, November 1981 Publicarions and Public RcJarions of Scare Agencies in Virginia, January 1982 Occuparional and Professional Regulatory Boards in Virginia, January 1982 The CETA Program Adminisrered by Virginia:s Balance-of-Srare Prime Sponsor, May 1982 Working Capical Funds in Virginia, June 1982 The Occuparional and Professional Regulatory Sysrem in Virginia, December 1982 lnrerim Reporr. Equity of Currenr Provisions for Allocaring Highway Consrrucrion Funds in Virginia, December 1982 Consolidarion of Office Space in rhe Roanoke Area, December 1982 Scaffing and Manpower Planning in rhe Deparrmenr of Highways and Transporrarion, January 1983 Consolidarion of Office Space in Norrhern Virginia, January 1983 lneerim Reporr. Local Mandares and Financial Resources, January 1983 Inrerim Reporr, OrganiZJlrion of rhe Execurive Branch, January 1983 The Economic Porenrial and Managemene of Virginia's Seafood lndusrry, January 1983 Follow-Up Reporr on rhe Virginia Deparrmene of Highways and Transporrarion, January 1983 1983 Reporr to rhe General Assembly, October 1983 The Virginia Division for Children, December 1983 The Virginia Division of Voluneeerism, December 1983 SCare Mandares on Local Governmenrs and Local Financial Resources, December 1983 An Assessmenr of Serucrural Targers in rhe Execurive Branch of Virginia, January 1984 An Assessmene of rhe Secrerarial Sysrem in rhe CommonweaJrh of Virginia, January 1984 An Assessmenr of rhe Roles of Boards and Commissions in rhe Commonwealrh of Virginia, January 1984 OrganiZJlrion of rhe Execurive Branch in Virginia, A Summary Reporr, January 1984 1984 Follow-up Reporr on rhe Virginia Deparrmenr of Highways and Transporrarion, January 1984 Inrerim Reporr. Ceneral and Regional Sraffing in rhe Deparrmene of Correcrions, May 1984 Equiry of Currenr Provisions for Allocaring Highway and Transporrarion Funds in Virginia, June 1984 Special Educarion in Virginia's Training Cenrers for rhe Meneally Rerarded, November 1984 Special Educarion in Virginia's Menral HeaJrh Faciliries, November 1984 Special Reporr, ADP Coneracring ar rhe Srare Corporarion Commission, November 1984 Special Reporr, The Virginia Srare Library's Conrracr Wirh The Compurer Company, November 1984 Special Reporr, The Virginia Tech Library Sysrem, November 1984 Inrerim Progress Reporr, Review of rhe Virginia Housing Developmene Aurhoriry, February 1985 Special Reporr. Parene and Copyrighr Issues in Virginia Srare Governmenr, March 1985 Virginia's Correcrional Sysrem, Popularion Forecasring and Capacity, April 1985 The Communiry Diversion Incenrive Program of rhe Virginia Deparrmenr of Correcrions, April 1985 Securiry Sraffing and Procedures in Virginia's Prisons, July 1985 Towns in Virginia, July 1985 Local Fiscal Seress and Srare Aid, A Follow-Up, August 1985 1985 Reporr to rhe General Assembly, September 1985 The Virginia Housing Developmene Aurhoriry, October 1985 Special Reporr. Cousreau Ocean Ceneer, January 1986 Sraff and Faciliry Urilizarion by rhe Deparrmene of Correcrional Educarion, February 1986 Cosrs for rhe Srandards of Qualiry - Parr J. Assessing SOQ Cosrs, February 1986 Joint Legislative Audit & Review Commission Suite 1100 General Assembly Building capitol Square Richmond, VIrginia 23219 (804) 786-1258