National Fish and Wildlife Foundation’s Support for Forest Management Research on Birds

An Evaluation of Outcomes and Impacts Evaluation Team

Daniel R. Petit, Ph.D. Dan Petit has 24 years of experience in forest bird research across three continents. NorthWoods Consultants He has been a Postdoctoral Research Fellow with the Smithsonian Environmental 360 South Miller Road Research Center in Edgewater, Maryland; National Nongame Bird Coordinator with Fairlawn, Ohio 44333 U.S. Fish and Wildlife Service in Washington, D.C.; National Coordinator of the 330-864-1227 Science Applications Program of the U.S. Geological Survey, Reston, Virginia; and Manager of Natural Resources Research, Cleveland Metroparks, Cleveland, Ohio. Dan received his M.S. in Zoology from the Ohio State University (1984), and Ph.D. in Zoology from University of Arkansas (1991).

Joseph A. McGlincy Joe McGlincy is Director of The Wildlife Company, an environmental consulting Certified Wildlife Biologist firm. He graduated from Auburn University (1975), and has over 27 years experi- The Wildlife Company ence in the wildlife and environmental fields. Joe worked for 14 years as a profes- 305 West Shotwell Street sional wildlife biologist for a large forest products company coordinating environ- Bainbridge, Georgia 39819 mental issue management and outdoor recreation business for over six million 229-246-5785 acres of industrial forestland. He currently specializes in endangered species consultation and the Sustainable Forestry Initiative certification.

Citation: Petit, D.R. and J.A. McGlincy. 2004. National Fish and Wildlife Foundation’s support for forest management research on birds: an evaluation of outcomes and impacts. Unpublished Final Report to the National Fish and Wildlife Foundation. EXECUTIVE SUMMARY

Executive Summary

National Fish and Wildlife Foundation’s Support for Forest Management Research on Birds

Evaluation of Outcomes and Impacts

Purpose • Communication of results was generally good, and many peer-reviewed technical reports were To evaluate the impact of National Fish and Wildlife published and communicated at professional Foundation-sponsored research on advances in meetings. Publications directly targeting forest forest bird management and conservation, and on managers often were lacking, which likely hin- forest bird populations and their habitats. dered field application of research results.

• Research results influenced forest management Project Overview policy and planning in a general sense and several significant changes in forest management Nine objectives associated with the Foundation’s plans and policies were made. forest bird research program were assessed for each of eleven North American research projects sup- • Several notable adjustments to local forest ported by the Foundation between 1991 and 1999. management actions resulted due to information In 2003, site visits, telephone interviews, and other originating from research projects, but results forms of data collection were used to document the usually did not translate into management action. outcomes of each project. Information was gathered from researchers, project collaborators, and pro- • Research validated the significance of several spective users of research results -- primarily forest existing forest management practices for forest products industry, nongovernmental organizations, birds. Anecdotal evidence suggests that changes and government agencies. made to forest management practices have benefited, or will benefit in the future, birds and their habitats, but lack of assessment and moni- Key Findings toring protocols by land managers prevented rigorous assessment of those possibilities. • The Foundation incorporated a diverse set of stakeholder interests into the forest research program and engaged highly qualified research Primary Recommendations teams to investigate priority issues. The widespread and positive outcomes of these • Foundation funds were used effectively to lever- select research projects suggest that the National age additional research funding, and to stimulate Fish and Wildlife Foundation should continue and numerous research projects on forest manage- enhance its support of forest management research ment and conservation. on birds and, possibly, other wildlife. The Program must better emphasize timely and effective commu- • Valuable and reliable information was derived nication of results to prospective users of research from supported projects, most of which was information through, for example, publication of directly applicable to forest management and results in appropriate formats and development of conservation. regional workshops aimed at synthesizing and disseminating available scientific information.

Table of Contents

Introduction...... 6

Program Objectives...... 8

Design and Methods...... 14

The Projects...... 18

Objective 1: Identify Information Needs...... 22

Objective 2: Support Quality Projects...... 26

Objective 3: Stimulate Additional Research...... 28

Objective 4: Generate Valuable Information...... 32

Objective 5: Disseminate Information Effectively...... 36

Objective 6: Incorporate Results Into Plans...... 40

Objective 7: Implement Management Actions...... 46

Objective 8: Habitat Gains...... 48

Objective 9: Bird Population Gains...... 50

Conclusions & Recommendations...... 52

Literature Cited...... 56 EVALUATION OF FOREST RESEARCH

INTRODUCTION

The main purpose of rogram evaluation is the pro- The mission of the Foundation is evaluation is to improve the Pcess of systematically collecting pursued through three principal quality of a program or information about a program to modes of action: (a) grants pro- project by identifying its assess whether acceptable stan- grams, (b) partnership develop- strengths and weaknesses. dards or expectations (such as ment, and (c) leveraging of funds. processes, objectives or outcomes) Between 1984 and 2002, the Dr. Murari Suvedi, Professor, have been met (Case et al. 1988, Foundation supported over 5,000 Michigan State University, NSF 1998). Evaluation is an essen- grants and leveraged $226 million in 2003 tial step in the evolution of a pro- federal funds for more than $700 gram, as it can offer an unbiased million in on-the-ground conserva- assessment of what works and what tion, education, and research. This does not work under existing has resulted in 27 million acres of programmatic protocols, and can restored and managed wildlife provide insight into potential habitat, more than 14 thousand changes necessary for processes or miles of restored rivers and streams, expectations. In essence, program new models of private land steward- evaluation is a form of organizational ship, and stronger educational “accountability,” the outcome of programs in schools and communi- which can be a more effective and ties (NFWF 2002, 2003). publicly trusted organization (Inde- pendent Sector 2000). The Neotropical Migratory Bird Program The National Fish and Wildlife Foundation By the 1980s, mounting evidence was beginning to paint an alarming The National Fish and Wildlife picture of the demise of neotropical Foundation (hereafter, Foundation) migratory birds, species that mi- is a private, non-profit, 501(c)(3) tax- grate annually to the Caribbean exempt organization established by Basin and Central and South the United States Congress in 1984 America from North American The Magnolia Warbler, a for the express purpose of protect- breeding grounds. In response to neotropical migrant, is one ing and enhancing fish, wildlife, and widespread concern about bird of more than 400 species plants, and the habitats upon which populations in the United States, the upon which Partners in those species depend. The mission Foundation, along with federal and Flight focuses its conserva- of the Foundation centers on state government agencies, nongov- tion efforts. While most promoting healthy populations of ernmental organizations, and private populations of Magnolia fish, wildlife, and plants by generat- industry, launched the Neotropical Warblers are secure, those ing new and unique partnerships Migratory Bird Conservation Pro- occupying interior portions between the public and private gram in 1990. The goal of the of North America have sectors and strategically investing in initiative, largely carried out under declined by more than 8% conservation and sustainable use of an umbrella public-private partner- per year during the past 37 natural resources. ship called Partners in Flight, was to years. support local, regional, national, and

6 INTRODUCTION international species and habitat Early in the development of the conservation and management Neotropical Migratory Bird Conser- efforts to benefit neotropical migra- vation Program, the Foundation tory birds. Partners in Flight, recognized that forestry-based through the guiding efforts of the research on non-hunted birds was Foundation, is now widely recog- scarce and that scientific informa- nized as one of the largest and tion in this area would be paramount most successful public-private to successful bird conservation. In conservation enterprises ever 1990, the Foundation established a established. Its conservation value forest bird research program has reached from the frozen taiga of (hereafter called the Program) Alaska, to the arid deserts of geared towards establishing a better Sonora, to the wet pampas of understanding of the habitat re- Numerous independent Argentina (Finch and Stangel 1993, quirements of forest birds and the activities and proposals Martin and Finch 1995, Bonney et potential impacts of various land have materialized..., but a al. 2000). In addition, the initiative uses, including silvicultural prac- complete, coordinated has stimulated the highest levels of tices, on those populations. The framework to ensure national governments to recognize focus of each supported research [neotropical] migrant the need for more proactive conser- project was intended to be related, protection was missing. vation of migratory birds (e.g., either directly or indirectly, to [Partners in Flight] is Executive Order 13186, Federal developing ways in which conserva- designed to address this Register 66:3853-3856, 2001). tion of forest-dwelling birds could be need by coordinating effectively integrated into existing cooperative efforts among forest management and land use, federal, state and local A Program for Forest Bird including silvicultural practices and government agencies, professional alliances, Research landscape- and regional-level planning. During the 1990s, more philanthropic foundations, and private companies. Effective conservation and manage- than $1.5 million was designated by the Foundation for forest manage- ment of forested habitats in North Dr. Deborah Finch, Research ment research on birds, which America were identified as principal Wildlife Biologist, USDA Forest focal areas of Partners in Flight additionally leveraged more than $4 Service, 1991 because many species of forest- million in partner contributions. dwelling birds had exhibited signifi- cant population declines during the preceding 30 years. The issue of Evaluating the Program forest management was seen as especially critical to the success of In 2003, the Foundation commis- migratory bird conservation because sioned a broad-based evaluation of more than 70% of forested lands in the effectiveness of the Program in the United States were owned by generating high quality research private individuals and corporations, information about forest birds, and much of which was used to some in applying that information to forest Additional information extent for timber production and management and conservation. The relating avian communities extraction (Wigley and Sweeney evaluation, while seeking to better to forest composition, 1993), or was susceptible to devel- understand the overall contributions distribution, fragmentation opment. Furthermore, the majority of the Program, was not intended to and various silvicultural of acreage in the federal and state be a comprehensive assessment of practices is needed to forest systems were open to timber all projects. Rather, it was designed ensure the future of sensi- or mineral extraction and other to use a select, but diverse, subset tive neotropical migratory multiple use designations, thereby of supported projects to examine the birds. potentially creating habitat detrimen- outcomes and impacts of various tal to the conservation of biological approaches to forest bird research Dr. James Dickson (Research diversity, including forest birds on conservation and management Wildlife Biologist, USDA Forest (Williams and Rinne 1992). activities (see Design and Methods). Service) and others 1993

7 EVALUATION OF FOREST RESEARCH

PROGRAM OBJECTIVES

A performance measure he Foundation’s Program Objectives, performance measures, is a means of judging Tdirector met with evaluators to and core indicators form a hierarchi- whether an objective has discuss the overall purpose and cal set of program benchmarks that been fulfilled. It can be objectives of the forest bird research indicate the level at which a pro- expressed in several ways initiative. From those discussions, gram is performing. Outlined below but, in general, a perfor- evaluators developed language to is the structured set of benchmarks mance measure is a reflect nine principal programmatic used for this Program evaluation. statement that reflects a objectives. Several of the objectives summation of activity or had not been formally articulated by effort that can be expressed the Foundation prior to that exer- in a quantitative or qualita- cise, although all were recognized to OBJECTIVE 1 tive manner. The activity or be integral to Program success. Identify priority information effort is deemed to be needs pertaining to forest bird critical for achieving a Objectives statements provided the management and conserva- particular objective. foundation upon which Program tion, and incorporate those performance could be evaluated. needs into Program A core indicator, on the Evaluators chose to evaluate other hand, represents a objectives (and, hence, Program performance) using an approach PERFORMANCE MEASURE 1.1 defined value, product, or Foundation shall seek input from characteristic that can be similar to that used by the United States government (Government leaders and principal stakeholders used to directly measure in bird and forest management the result of the activity or Performance and Results Act of 1993; GPRA) and the Sustainable and conservation to identify priority effort identified through a information needs. performance measure. By Forestry Initiative (SFI) Program of the Sustainable Forestry Board. summarizing information CORE INDICATORS about core indicators, Both institutions have endorsed the concept of performance measures 1.1.1 Incorporation of state-of- performance measures can the-art information, including be assessed. and core indicators (OMB 1995, Sustainable Forestry Board 2002) to refereed literature and organiza- assess the level of achievement of tional and agency management program objectives. [See sidebars and policy documents, into on this page for definitions.] identification of priority informa- tion needs. For each of the nine objectives, one or more performance measures 1.1.2 Participation of Founda- were identified and used to assess tion staff in formal meetings the extent to which an objective had involving principal stakeholders been met. Core indicators were in forest bird management and used as the metrics by which conservation. performance measures were directly evaluated. For objectives to be 1.1.3 Obtain prior to Program fulfilled, performance measures implementation, and at regular needed to be fully supported by intervals thereafter, external core indicator metrics. review of priority information needs.

8 PROGRAM OBJECTIVES

PERFORMANCE MEASURE 1.2 CORE INDICATORS Objectives 1 and 2 pertain Foundation shall incorporate high 2.1.1 Availability of funding and to the effectiveness of the priority information needs from a partnership opportunities are Foundation in identifying diverse set of principal stakehold- broadly and strategically dis- important information needs ers into priority information needs. seminated via published, elec- and funding research tronic, and verbal announce- projects that are capable of CORE INDICATORS ments. delivering high quality 1.2.1 Stated priority Program information about those information needs reflect high 2.1.2 Adequate numbers of issues. This type of evalua- priority information needs of a internationally, nationally, and tion is termed “process diverse set of principal stake- regionally recognized research- evaluation” (see page 14). holders. ers in forest bird management and conservation apply for PERFORMANCE MEASURE 1.3 funding, or seek partnership Foundation shall incorporate high support, from the Foundation. priority information needs from a diverse set of principal stakehold- PERFORMANCE MEASURE 2.2 ers into Program funding deci- Foundation shall ensure that sions. adequate funding and partnership support is available to implement CORE INDICATORS research projects addressing 1.3.1 Funded research projects priority information needs. reflect high priority information needs of a diverse set of princi- CORE INDICATORS pal stakeholders. 2.2.1 Adequate Foundation funding is available to support high priority information needs.

2.2.2 Adequate partnership development is available to implement projects addressing high priority information needs.

PERFORMANCE MEASURE 2.3 Research proposals funded, or developed through partnerships, by the Foundation are highly likely to produce useful and reliable OBJECTIVE 2 scientific information. Provide adequate financial CORE INDICATORS Foundation-sponsored and partnership support for 2.3.1 Principal investigators and research led by Richard research projects that are collaborators are recognized Lancia of North Carolina likely to successfully authorities with proven track State University found that address priority Program records on forest bird manage- the Downy Woodpecker information needs ment and conservation research. (above) and Eastern Titmouse (left), both PERFORMANCE MEASURE 2.1 2.3.2 Supported research nonmigratory cavity-nesting Foundation shall encourage projects contain strong experi- species, were not affected application for funding and part- mental designs, justifiable by forest fragmentation that nerships from internationally, timetables for project completion, resulted from timber har- nationally, and regionally recog- and realistic expectations for vesting practices in the nized researchers in forest bird addressing priority information coastal plain of South management and conservation. needs. Carolina.

9 EVALUATION OF FOREST RESEARCH

Objectives 3-6 pertain to the OBJECTIVE 3 short-term objectives of the Stimulate additional avenues Program, objectives that of research and areas of typically can be assessed inquiry on forest bird manage- through relatively straightfor- ment and conservation ward activities and prod- ucts. This type of evaluation PERFORMANCE MEASURE 3.1 is termed “outcome evalua- Principal investigators, collabora- tion” (see page 15). tors, or other researchers use information derived from supported projects to further investigate priority Program information and related needs. OBJECTIVE 4 Data and information derived CORE INDICATORS from supported research 3.1.1 Projects with multiple projects are of high value to years of Foundation support land managers, planners, and expand upon research and other other decision-makers objectives during latter years of support. PERFORMANCE MEASURE 4.1 Stated research goals and objec- 3.1.2 Results from supported tives of supported projects are projects are incorporated into achieved. research needs assessments of agencies, organizations, and CORE INDICATORS conservation partnerships. 4.1.1 Research hypotheses and questions posed in proposals are 3.1.3 Results from supported effectively addressed. projects stimulate other investi- gations. 4.1.2 Proposed project products are satisfactorily developed. 3.1.4 Results from supported projects are referenced in other PERFORMANCE MEASURE 4.2 research proposals. Definitive results are derived from supported projects.

CORE INDICATORS 4.2.1 Results from supported projects are clear with respect to In a study supported by the Predation Rates (%) on addressing stated research Foundation, Thomas Martin Nests of Black-and-White questions. of the University of Arkansas Warblers Under Different found that the Black-and- Silvicultural Treatments PERFORMANCE MEASURE 4.3 White Warbler (photo Project results have relatively direct above), a neotropical application to forest management migrant, suffered greater and conservation. nest predation in stands that had been commercially CORE INDICATORS thinned compared to that in 4.3.1 Results from supported closed canopy forests and projects can be used immedi- mature stands where trees ately and directly by practitioners were selectively removed of forest management and (Barber et al. 2001). conservation.

10 PROGRAM OBJECTIVES

OBJECTIVE 5 Black-throated Information from supported Green Warbler research is effectively commu- nicated to prospective users

PERFORMANCE MEASURE 5.1 Information derived from research projects is communicated by principal investigators to prospec- tive users in a timely and effective manner. Black-capped Chickadee CORE INDICATORS PERFORMANCE MEASURE 5.2 5.1.1 Project results are formally Information derived from research communicated at professional, projects is communicated by project technical meetings. collaborators to prospective users in a timely and effective manner. 5.1.2 Project results are formally communicated at meetings with CORE INDICATORS professional practitioners of 5.2.1 Project results are formally forest management and conser- communicated at meetings with Common vation. professional practitioners of Yellowthroat forest management and conser- 5.1.3 Project results are formally vation. communicated at meetings with national or regional conservation 5.2.2 Project results are formally or management groups. communicated at meetings with national or regional conservation 5.1.4 Project results are formally or management groups. communicated at meetings with local conservation or manage- 5.2.3 Project results are formally John Hagan and colleagues ment groups. communicated at meetings with at the Manomet Center for local conservation or manage- Conservation Sciences 5.1.5 Project results are pub- ment groups. documented habitat prefer- lished in refereed technical ences (shown here as birds journals, symposium proceed- 5.2.4 Project results are pub- per survey plot) of forest- ings, or books. lished in non-refereed publica- dwelling birds on industrial tions, such as technical reports, timber lands in Maine. Most 5.1.6 Project results are pub- brochures, magazines, newslet- species showed clear lished in non-refereed publica- ters, or the WWW. preferences for mid- to late- tions, such as technical reports, rotational stands (e.g., brochures, magazines, newslet- PERFORMANCE MEASURE 5.3 Black-throated Green ters, or the WWW. Information derived from supported Warbler; top graph and research projects is communicated photo) or for early succes- by the Foundation to prospective sional stands (Common users in a timely and effective Yellowthroat; bottom graph). manner. Several species, however, showed little preference for CORE INDICATORS a specific age class of forest 5.3.1 Project results are formally (Black-capped Chickadee; communicated at meetings with middle graph, bottom national or regional conservation photo). Data taken from or management groups. Hagan et al. 1997.

11 EVALUATION OF FOREST RESEARCH

Objectives 7-9 are associ- OBJECTIVE 6 PERFORMANCE MEASURE 6.2 ated with long-term out- Information from supported Information and recommendations comes of great conse- research projects provides from research projects are incorpo- quence, such as positive management and rated into land use and species changes in the behavior of conservation guidance to conservation and management plans. people or in the population prospective users status of a target species. CORE INDICATORS While often difficult to PERFORMANCE MEASURE 6.1 document, achievement of 6.2.1 Information from projects Information and recommendations is incorporated into private and these types of outcomes from research projects are incorpo- ultimately define the suc- non-governmental land use and rated into organizational policies, species conservation and cess of a program. This type procedures, and practices. of evaluation is termed management plans. “impact evaluation” (see CORE INDICATORS page 15). 6.2.2 Information from projects 6.1.1 Information from projects is incorporated into government is included in private and non- land use and species conserva- governmental organizations’ tion and management plans. policies, procedures, and practices.

6.1.2 Information from projects OBJECTIVE 7 is included in federal, state, or Recommendations derived local government policies, from supported research procedures, and practices. projects are implemented by private, nongovernment, and 6.1.3 Information from projects government groups is included in the policies, procedures, and practices of PERFORMANCE MEASURE 7.1 professional societies. Management and conservation recommendations derived from projects are implemented by private and nongovernmental organiza- The Ovenbird, a neotropical migrant tions, and government agencies. considered to be sensitive to forest fragmentation and disturbance, was CORE INDICATORS studied in four 7.1.1 Private, nongovernmental, Foundation- and government groups imple- sponsored ment forestry and silvicultural projects across practices that are based upon the United project results. States. In those investigations, 7.1.2 Approaches to land use Ovenbirds planning and land acquisition exhibited a that are based upon project mixed response results are implemented by to silvicultural private, nongovernmental, and practices, government groups. suggesting that the level of 7.1.3 Private, nongovernmental, impact on and government groups imple- Ovenbirds depends upon the type of ment species management and local forest disturbance and the conservation practices that are landscape context within which those based upon project results. disturbances take place.

12 PROGRAM OBJECTIVES

OBJECTIVE 8 Ovenbird Responses (Birds/100 ha) Habitat conditions are to Silvicultural Treatments in the improved for Missouri Ozarks. forest-dwelling birds Research led by PERFORMANCE MEASURE 8.1 John Faaborg of Implementation of management and the University of conservation practices based upon Missouri examined project information directly improves changes in abun- habitat conditions for forest-dwelling dance of Ovenbirds birds. after selective cutting (single tree CORE INDICATORS and small group) 8.1.1 Higher quality forest lands and clearcutting in are acquired or protected based 1996 (Gram et al. upon project information. 2003). Ovenbirds declined signifi- 8.1.2 Forest acreage is directly cantly after both improved through management types of harvesting or restoration practices that are and, notably, implemented based on project Ovenbirds even information. OBJECTIVE 9 declined in the Population status of forest- adjacent control dwelling birds is improved plots (uncut mature forest). Acadian PERFORMANCE MEASURE 9.1 Flycatchers (photo Implementation of management and at left) exhibited conservation practices based upon similar patterns of project information improves popula- decline. Faaborg tion status of forest-dwelling birds. and his colleagues suggested that CORE INDICATORS birds might choose 9.1.1 Local populations of breeding habitats forest-dwelling birds exhibit more based upon positive population trajectories environmental after implementation of manage- features well ment and conservation practices beyond the immedi- derived from supported projects. ate nesting territory or forest stand. 9.1.2 Regional populations of forest-dwelling birds exhibit more positive population trajectories after implementation of manage- ment and conservation practices derived from supported projects.

Acadian Flycatchers require rela- 9.1.3 Increased reproductive tively undisturbed, closed-canopied success or survival, or other forests during both the breeding indicators of population status, of season in the United States and forest-dwelling birds occurs after during the overwintering period in implementation of management Central America. and conservation practices derived from supported projects.

13 EVALUATION OF FOREST RESEARCH

DESIGN AND METHODS

The principal goal of this solid underlying conceptual diverse collection of research evaluation was to examine Aand practical design was issues, geographic locations, the relative successes of paramount to extracting accurate organizational affiliations, and various approaches to and useful information about the approaches to project implementa- research on forest bird Program (for general rationale, see tion. Hence, the evaluation was not conservation and manage- Benson and Michael 1987, Suvedi intended to assess the overall ment activities, and to 2003). The approach used was success of the Program. Instead, it identify potential factors that discussed and mutually agreed used a select group of projects to may have limited those upon by the Foundation and (a) elucidate strategies that led to successes. Program evaluators prior to imple- conservation successes, and (b) mentation. Those a priori discus- identify potential impediments to sions allowed both Program admin- those successes. istrators and evaluators to define, understand, and agree upon the By focusing on this non-random specific information that was neces- subset of projects, the evaluation sary to evaluate the Program. was able to address a primary Discussions also provided the interest of the Program: developing opportunity to choose the most effective means of generating and appropriate, realistic, and cost- disseminating scientific information, effective means of collecting the and incorporating that information necessary information. into on-the-ground management and conservation activities.

Goal of the Evaluation Approach to Evaluation The principal goal of this evaluation was to examine the relative suc- This project employed three basic cesses of various approaches to types of evaluation to provide a research on forest bird conservation comprehensive assessment of and management activities, and to Program strengths and weak- identify potential factors that may nesses: (a) process evaluation, (b) have limited those successes. outcome evaluation, and (c) impact evaluation (McNamara 1998, NSF As such, the evaluation was built 1998). upon assessment of 11 research- based projects identified a priori by Process evaluation assesses the the Foundation. These projects did extent to which a program is operat- not represent a random subset of all ing as intended, and examines such research projects funded by the issues as engaging target audi- Foundation. Rather, the evaluation ences and effectiveness of decision- included some of the largest making. Because the process by projects (based upon length and which the Foundation made deci- extent of funding) that the Program sions and managed day-to-day supported and represented a operations of the Program was not

14 DESIGN AND METHODS the primary focus of this evaluation, questions were aimed at generating the results presented here offer only information that reflected the extent a cursory treatment of the effective- to which core indicators had been ness of Program management. achieved. Nevertheless, while offering less than a detailed, comprehensive Evaluators quickly realized, how- examination of all management ever, that the closed-ended survey issues, inclusion of process evalua- questions that largely dominated the tion still provided valuable insight survey instrument would not be into several major aspects of totally effective in extracting some Foundation decision-making as it relevant information from respon- related to Program outcomes and dents. Closed-ended survey ques- impacts. tions (Babbie 1995, Fink 1995) are characterized by a set of questions, Outcome evaluation reviews the the range of answers to which are A 44-page survey instru- relationship between the intended, predetermined by evaluators. ment was developed to help relatively short-term objectives of a Hence, little opportunity is given to standardize presentation of program and the actual, docu- respondents to express supplemen- questions and documenta- mented outcomes. Such evalua- tal feelings and attitudes about focal tion of responses. The tions often focus on fundamental issues. While closed-ended survey complete survey instrument products or changes anticipated designs have many beneficial is included as a separate from program implementation. properties, the limited number of attachment to this report. projects used and the lack of pre- Impact evaluation is a type of existing information on the issues outcome evaluation that focuses on that were likely to be encountered broad, longer-term programmatic by evaluators in this evaluation, impacts, such as changes in human required an additional, alternative behavior or long-term population approach to information collection. trajectories of wildlife. A naturalistic design was also incorporated into interviews to Approach to Data Collection generate more information per respondent. This approach used a In addition to integrating multiple more informal, ad hoc series of approaches to program evaluation, open-ended questions (Babbie Program evaluators also employed 1995, Fink 1995) that built upon several means of information some of the closed-ended questions collection and analysis, including posed through the survey instru- survey and naturalistic designs ment. Naturalistic surveys and (Wilde and Sockey 1995). open-ended questions can be prone to evaluator bias and are often less A survey design uses a standard- quantitative and conducive to ized set of questions that is used by rigorous analysis compared to all evaluators to collect either closed-ended survey instruments, qualitative or quantitative data. but can be effective when relatively Potential answers to questions often few subjects from which to collect are limited by the evaluator and are information are involved and supple- categorized to assist in quantitative mental information about the survey summarization of results. population or focal issues are of value (Fink 1995). A 44-page survey instrument was developed to help standardize Hence, evaluators used an adaptive presentation of questions and approach to collection of survey documentation of responses. All data, initially favoring a closed-

15 EVALUATION OF FOREST RESEARCH

ended series of standardized information; and the need for questions, but (after assessing the additional independent clarification content of initial interviews) later of certain aspects related to the incorporated a more naturalistic value of research projects. In total, design that stressed open-ended approximately 100 individuals were questions and collection of supple- interviewed. mental information to clarify and expand upon answers to the more Interviews typically were informal. structured survey questions. With The evaluator generated initial collective adjustment of behavior, questions from the survey instru- evaluators were able to adapt to ment, but allowed the individual forest bird research and manage- being interviewed full and unim- ment issues that were not antici- peded expression of views about the pated prior to implementation of the research project under investigation. survey. When relevant, evaluators often requested specific written materials from respondents pertaining to their Data Collection Methods involvement in, or use of information from, the research project. Information about the Program and its supported projects was collected Interviews required 20 minutes to 4 between January and November hours depending upon the role of 2003 using a combination of site each individual in the research, the visits; face-to-face, telephone and e- individual’s familiarity with the mail interviews; and literature project, or the extent to which the reviews. Interviews focused princi- individual made use of the research pally on four groups of individuals: results. At the time of each inter- view, evaluators wrote answers to „ Principal investigators of re- questions directly on the survey search projects instrument or in a notebook. Re- spondents were not shown the „ Collaborating scientists and land information recorded by evaluators. managers Visits were made to about half of the field research sites to obtain a better „ Users of research information, understanding of research design, including government and results, and conservation implica- nongovernmental groups, and tions. private landowners

„ Foundation Program director Assessment of Program Objectives Both evaluators had responsibility for primary assessment of 5 or 6 The extent to which each of the nine research projects, although each Program objectives were fulfilled individual reviewed proposals, was based upon careful scrutiny of reports, and other supporting information gathered in support of materials from all projects. Selection core indicators and performance of individuals to interview was based measures. Both evaluators indepen- upon recommendations of principal dently assessed the results of their investigators; availability of local, respective surveys for each Program regional, and national leaders in objective, then collectively dis- forest conservation, management, cussed those independent evalua- and research; identification of tions to reach a final result. For prospective users of research virtually all nine Program objectives,

16 DESIGN AND METHODS independent assessment of results For this evaluation, assessment of were exceptionally similar, leading to each Program objective was sum- little disagreement in the final marized on a subjective, 4-point assessment by the two evaluators. relative scale (see figure below): The similarity of these independent conclusions, while only representing Poor: Performance measures the views of two individuals, lends consistently unfulfilled (more than support to the credibility of conclu- two-thirds of projects did not fulfill sions. measures).

Results of this evaluation are Fair: Inconsistent attainment of presented both as quantitative and performance measures (between qualitative narratives of principal one-third and two-thirds of projects conclusions and highlights for each did fulfill measures). objective. However, because of the open-ended nature of many of the Good: Performance measures questions posed to respondents, consistently fulfilled (more than two- quantitative evaluation of overall thirds of projects did fulfill mea- Program objectives was not fea- sures). sible. Furthermore, because no “standards” exist for assessing the Excellent: Program or project relative success of forest bird performance consistently exceeded research, even collection of detailed, expectations (more than two-thirds quantitative assessment data for of projects did fulfill measures and Program objectives would still have exceeded expectation of evaluators). required subjective assignment of overall scores of success for the nine objectives.

FAIR EXCELLENT

POOR GOOD

Summary scale used to depict overall attainment of each of the nine Program objectives. Each objective received one score based upon overall strength of associated performance measures.

17 EVALUATION OF FOREST RESEARCH

THE PROJECTS

Number of Projects leven projects were chosen by The $1.5 million contributed by the Receiving Various Levels Ethe Foundation for inclusion in Foundation leveraged at least $4.1 of Foundation Funding the Program evaluation. All projects million in matching (federal and were focused on applied research non-federal) monies. This latter aimed at better understanding the estimate, however, is likely to be habitat requirements of migratory conservative, as not all projects birds breeding in North American carefully tracked and reported funds forests, and had been completed at originating in host organizations or least 4 years prior to evaluation to from in-kind contributions donated ensure adequate time for analysis by collaborators. Nevertheless, for Thousands of Dollars and dissemination of results. every federal dollar the Foundation directed towards these research Nongovernmental organizations projects, at least $3 in direct fund- comprised more than half (6) of the ing, materials, or services were institutions to which research secured. awards were made. Four university faculty received grants, as did one Research projects were conducted state agency biologist. Eight of the in a diverse set of forest types and principal investigators or primary ecoregions of North America. In fact, contacts had earned doctoral the 11 projects have direct applica- degrees. All projects represented tion to at least 18 ecoregions (Bailey collaborative efforts with at least two 1995) of the United States and additional organizations. Canada.

All projects were supported for 1-6 The study design of seven of the years between 1991 and 1999. The projects were closely tied to active average project received 3.4 years forest management, in particular of financial support from the Foun- timber harvesting practices. In dation. Several projects were able to addition, nearly all projects incorpo- expend funds for longer periods rated a landscape-level perspective than originally planned due to into the implementation and inter- adaptation of project timetables to pretation of the investigation. Number of Projects available funding. Receiving Various Years of The objectives and focus of many of Support from Foundation A total of approximately $1.5 million the projects evolved over time of federal funding was directed by because of scientific knowledge the Foundation to the 11 projects, gained during preceding years. The an average of about $140,000 per following pages provide a brief project, which was relatively large synopsis of the objectives of each compared to typical awards the project used in this evaluation. Foundation makes for bird conser- Project titles reflect the general vation work. Total funding ranged thrust of each project and not between approximately $26,000 necessarily actual project titles Years of Support and $450,000 for each project. identified by principal investigators.

18 THE PROJECTS

LOCATIONS OF ELEVEN PROJECTS USED FOR PROGRAM EVALUATION*

Dr. Rex Sallabanks Sustainable Ecosystems Institute Forestry and ecosystem management Dr. Patricia Heglund Turnstone Ecological Research Associates Monitoring and habitat requirements Ms. Lee Pfannmuller Minnesota Department of Natural Resources Monitoring and habitat requirements

Dr. Susan Hannon University of Alberta Forest fragmentation and reserve design Dr. John Hagan Manomet Center for Conservation Sciences Forestry practices

Mr. Bob Altman Avifauna Northwest Salvage logging

Dr. Richard Lancia North Carolina State University Forested corridors

Dr. John Faaborg University of Missouri Forestry practices Dr. T. Bently Wigley National Council for Air & Stream Improvement Landscape-level models of forest suitability Dr. Thomas Martin University of Arkansas Mr. Robert Ford Forestry practices Tennessee Conservation League Regional conservation planning

* Key: Principal Investigator or Contact Organizational Affiliation Focus of Research Project

19 EVALUATION OF FOREST RESEARCH

Silvicultural Practices and the Integrating Ecological and Health of Bird Populations Economic Goals into Industrial Forest Management Thomas Martin University of Arkansas & John Hagan University of Montana Manomet Center for Conservation Sciences „ Assess effects of silvicultural practices on breeding productiv- „ Develop models that integrate ity and abundance of birds in both economics and ecology in hardwood forests of Arkansas forest management

„ Identify habitat requirements „ Develop GIS-based landscape management tools

The pine-hardwoods forests „ Identify habitat requirements of the Ouachita Mountains of central Arkansas served „ Develop and facilitate stake- as an important study area holder discussion groups on for Foundation-supported forest management in Maine research on neotropical Regional Forest Changes and migratory birds. Populations of Forest Birds

Lee Pfannmuller Minnesota Department of Natural Resources

„ Develop long-term regional monitoring protocols for forest birds in Minnesota

„ Develop GIS-based landscape management tools Clearcut and corridor in Alberta’s aspen forests

Effects of Forest Fragmentation Resulting from Timber Harvest Ecoregional Planning Susan Hannon Robert Ford & Daryl Durham University of Alberta Tennessee Conservation League „ Assess effects of silvicultural „ Develop GIS-based habitat and practices on breeding birds in bird management tools aspen forests of Alberta

„ Develop and implement „ Evaluate benefits of movement ecoregional and inter-state bird corridors linking forest fragments conservation plans „ Develop recommendations for Birch-aspen forest in „ Enhance outreach and education spatial and temporal design of Minnesota efforts for bird conservation timber harvesting operations

20 THE PROJECTS

Effects of Salvage Logging on Effects of Timber Management Migratory Bird Populations Practices on Bird Populations

Bob Altman John Faaborg Avifauna Northwest University of Missouri

„ Evaluate effects of salvage „ Assess habitat use, reproductive logging practices on landbirds output, and abundance of birds breeding in lodgepole pine breeding in Missouri oak-hickory forests of Oregon forests managed under different silvicultural treatments

Timber Harvest, Forest Development of Landscape- Fragmentation and the Health level Models for Sustainable of Bird Populations Forest Management Uncut 50-70 year-old oak- hickory stands served as Rex Sallabanks T. Bently Wigley control plots in a study in Sustainable Ecosystems Institute & National Council of the Missouri Ozarks. Conservation Research Foundation the Paper Industry for Air and Stream Improvement

„ Assess effects of silvicultural practices and forest fragmenta- „ Identify habitat requirements of tion on productivity of breeding birds, reptiles, and amphibians in birds in western forests coastal forests of South Carolina

„ Develop mathematical models „ Better understand the regional dynamics of bird populations that incorporate economics and ecological factors in decision- making on industrial timberlands

The practice of clearcutting and its impact on bird populations was examined in most of the research projects

Habitat and Corridor Use by Assessing Bird Populations Birds in Industrial Forests and Habitat Use on Industrial Timberlands Richard Lancia North Carolina State University Patricia Heglund Turnstone Ecological „ Identify habitat and landscape- Research Associates Bottomland hardwood level requirements of birds forest, such as that found in breeding in coastal plain forests „ Identify habitat requirements the Tennessee National Wildlife Refuge, was a focus „ Examine the effects of forested „ Develop predictive models of of regional conservation corridors on birds in industrial habitat use and effects of planning in southeastern forests of South Carolina silvicultural treatments United States.

21 EVALUATION OF FOREST RESEARCH

OBJECTIVE 1 IDENTIFY INFORMATION NEEDS Identify priority information needs ecognition of the critical needs Creation of Working Groups pertaining to forest Rof constituents or customers is bird management an indispensable step in develop- Through the Partners in Flight and conservation, ment of any effective program. In network, the Foundation and its and incorporate those 1990, when the Foundation led partners established a formal needs into Program development of the Neotropical Research Working Group charged Migratory Bird Conservation Pro- with summarizing existing conserva- gram, “agencies and cooperators tion-based knowledge of migratory lack[ed] sufficient information on the bird populations and identifying population status and causes of additional research that was re- population changes of neotropical quired for implementing productive migrants to effectively conserve their management and conservation populations...” (Finch 1991:21). To actions. The Research Working address this gap in knowledge, the Group was comprised of a diverse Foundation initiated a five-pronged set of biologists from federal and approach for identifying priority state agencies, nongovernmental information needs of land manag- organizations, and industry, espe- ers, conservationists, and other cially those representing forest stakeholders. products. The Research Working The National Fish and Group continues as an active Wildlife Foundation used association today, some 13 years five means of identifying Use of Existing Information later. research and information needs for management and Prior to 1989, relatively few scientific Several valuable products were conservation of forest- studies had been conducted in the developed by the Research Working dwelling birds: Western Hemisphere that ad- Group within several years of the dressed issues of forest bird man- formative meeting of Partners in „ Use of existing scientific agement and conservation, espe- Flight, and those efforts helped the information cially those involving non-hunted Foundation establish a clear direc- species. That began to change, tion for its forest bird research „ Creation of research however, with a 1989 symposium program. In 1992, the group hosted and monitoring working and subsequent publication of a major workshop, “Status and groups comprised of proceedings entitled, Ecology and Management of Neotropical Migra- diverse stakeholders Conservation of Neotropical tory Birds,” that attracted more than and interests Migrant Landbirds (Hagan and 700 participants to Estes Park, Johnston 1992). That collection of Colorado. The workshop brought „ Guidance by federal research, along with several other together researchers and natural agencies notable works (e.g., Robbins et al. resources managers to discuss 1989, Terborgh 1989) provided the novel approaches to the conserva- „ Guidance by paper starting point for Foundation staff to tion and management of migratory products industry identify information gaps pertaining birds. One year later, a comprehen- to migratory bird conservation and sive review of neotropical migratory „ Guidance by to develop a strategy for overcoming bird ecology and management was conservation groups those limitations. published that summarized the

22 OBJECTIVE 1: IDENTIFY INFORMATION NEEDS presentations and outcomes of the Finally, the efforts of the Foundation Estes Park workshop (Finch and through Partners in Flight also Stangel 1993), including recom- stimulated identification of monitor- mendations for future research. A ing and assessment needs of forest companion volume was published birds in the United States (Butcher by Oxford University Press in 1995. et al. 1992) and in Latin America and the Caribbean (Rivera-Milan et al. 1994).

In sum, creation of working groups within the Partners in Flight network resulted in several established processes and databases that has allowed the Foundation and others to identify significant information gaps for management of forest habitats. This has been a productive approach to identification of informa- tion needs, as it stresses integration of a broad and cross-cutting set of perspectives in a noncontentious environment.

Guidance by Agencies

Approximately one quarter of the Several additional efforts under the United States land base is managed Partners in Flight umbrella were by the federal government. Because also directed at identifying critical much of those lands are forested forest bird research needs. The and because of their legislative Research Working Group imple- responsibilities relating to migratory mented a detailed survey of re- birds, the Foundation made con- search needs identified by land certed initial efforts to seek guidance managers, researchers, and others on critical information needs from involved in conservation of migratory several federal land management birds. That process, initiated in agencies, including the USDA 1995 but only recently published Forest Service, Bureau of Land (Donovan et al. 2002) and distrib- Management, and U.S. Fish and The Foundation was uted, has resulted in development Wildlife Service. instrumental in establishing of an interactive database on the several working groups WWW that provides detailed Agencies typically interacted with charged with identifying research needs for migratory Foundation staff through their research and information landbirds in North America. respective national offices, although needs for more effective information passed to the Founda- forest management. Several The Washington-Oregon chapter of tion was aggregated from regional resulting publications, Partners in Flight sponsored a more and field offices as well. Those including books edited by directed survey of research and agencies contributed both formal Martin and Finch (1995; information needs of public and and informal feedback on the above) and Finch and private land managers in the Pacific direction of the Foundation’s forest Stangel (1993; above left) Northwest (Arnett and Sallabanks bird research program (Program). summarized state-of-the-art 1998). That compilation of needs is In addition, federal biologists served knowledge about migratory helping guide forest research in as reviewers of research proposals landbird ecology and western forests. submitted to the Program. conservation.

23 EVALUATION OF FOREST RESEARCH

Migratory bird conservation... A major contribution to identification Second, through those professional of research needs was made by the contacts, Program staff were ...is only limited by our Forest Service through a detailed instrumental in creation of an knowledge of the species. synopsis of the ecology, habitat Industry Committee within the requirements, and conservation of Partners in Flight network. The GIS-based models are neotropical migratory birds prepared purpose of this working group was critical to an intelligent in support of the Neotropical Migra- to contribute to the search for viable approach to bird habitat tory Bird Conservation Program alternatives for the conservation of conservation. (Finch 1991). Information gaps migratory birds. Today, 13 forest necessary for effective management products corporations are active Dr. Steven Lewis and conservation were highlighted members of the Industry Commit- Nongame Bird Coordinator in that document which, in turn, tee, and have made significant U.S. Fish & Wildlife Service were referenced regularly by Pro- contributions to identification of gram staff in drafting funding important gaps in research. recommendations. Finally, Program staff developed a collaborative relationship with the Guidance by Industry National Council of the Paper Industry for Air and Stream Improve- Approximately one third of the ment, Inc. (NCASI), a partnership of United States is forested. Of those the paper products industry charged forested lands, 42% is managed by (in part) with developing guidance federal and state governments; the for sound management of forests remainder (58%) is privately owned and forest resources. NCASI offered (Smith et al. 2001). The Foundation another independent assessment of recognized that those private lands the needs for forest research. were critical for conservation of migratory birds and charted a course for inclusion of private Guidance by Conservation landholders in the Partners in Flight Groups initiative. In particular, the forest products industry, which currently Conservation-based organizations owns 9% of all forested lands in the have always played a pivotal role in United States (Smith et al. 2001) many aspects of Partners in Flight, and which actively manages the and identification of research needs bulk of its forests, was invited to for forest birds was no exception. A participate in identification of forest Nongovernmental Committee was management information needs. established within Partners in Flight and this group contributed to the Three approaches were employed overall direction of forest bird by Program staff to engage the conservation and research. forest products industry in Partners in Flight and identification of forest Program staff also consulted directly management information gaps. with biologists from those nongov- First, Program staff made direct ernmental groups. For example, a contact with administrators within program manager and professional the industry. For example, a profes- ornithologist from the National sional relationship was developed Audubon Society were regularly with the Director of Wildlife Re- consulted during the early years of sources at Champion International the Program. In latter years, the Corporation, which offered the American Bird Conservancy contrib- Program an intimate understanding uted to Program direction. of the needs and perspectives of the forest products industry.

24 OBJECTIVE 1: IDENTIFY INFORMATION NEEDS

Justification for Areas of Program Emphasis

Research directed at the conserva- tion of forest-dwelling migratory birds in North America could have taken numerous paths. The Pro- gram, as characterized by the 11 Manomet is doing what the research projects used in this evaluation, community should be doing. identified a subset of the important issues and disbursed the bulk of Donald Mansius funding towards research projects Director of Forest Policy and Management that addressed those information Maine Forest Service needs. While not the only areas of research investigated, three princi- pal avenues of research were stressed by the Program:

„ Effects of silvicultural practices on bird abundance and fecundity

„ Influence of landscape-level characteristics on bird abundance and distribution within landscapes dominated by timber harvest Evaluation of Objective 1 „ Development of predictive models of habitat use Identification of Important Information Needs Given the state of knowledge on these versus other better Identification of information needs that would best understood issues (e.g., population facilitate the conservation and recovery of migratory trends of migratory birds, forest bird populations was a critical first step in develop- fragmentation in agricultural ment of the Foundation’s Forest Bird Research Pro- landscapes) during the early 1990s, gram. The Program’s performance in meeting this the Foundation was justified in objective reflected a visionary process that stressed pursuing research directed at birds (a) incorporation of a diverse set of perspectives, (b) inhabiting “working” forests. This thrust was consistent with existing establishment of permanent working groups that priorities of industry (e.g., McMahon could offer long-term guidance, and (c) focus on a 1992), agencies (Kessler et al. restricted set of issues that, through examination in 1992), and conservation multiple geographic areas, could collectively contrib- organizations (e.g., IUCN et al. ute credible, broad-based recommendations for forest 1991). In addition, this evaluation management. confirmed the importance of the Program direction to principal stakeholders of forest sustainability and bird conservation. EXCELLENT

25 EVALUATION OF FOREST RESEARCH

OBJECTIVE 2 SUPPORT QUALITY PROJECTS

Provide adequate financial and imply disbursing funds to Finally, Program staff made presen- partnership support Sprojects is not a true measure of tations at meetings of Partners in for research projects success for granting agencies or Flight announcing funding availabil- foundations. Rather, the challenge ity and identifying priority research that are likely to of grantmaking is to identify and needs. With the exception of the successfully address support those projects that have a first two years, requests for Program priority Program high probability of achieving stated funding has always exceeded information needs goals and contributing to the resources available to fulfill those mission of the granting organization requests. Fewer than half of all (Orosz et al. 2003). The Foundation submissions typically are approved. used three means of soliciting proposals from prospective research teams who could make meaningful Project Selection contributions to its mission. The Program strongly favored First, Program staff contacted and proposals that were (a) focused on worked directly with nationally priority forest management issues, recognized scientists experienced in (b) based upon collaborative ecological research on forest birds partnerships with forest products to develop projects addressing high industry, (c) led by respected priority issues. scientists, and (d) underpinned by sound experimental design. While Second, the Foundation distributed those are credible components of requests for proposals for forest- decision-making, the Foundation based, ecological research. The was under no legislative or other mode of distribution included mandate to select projects for Partners in Flight newsletters, funding in a particular way. Thus, internet, web pages, and other the Program developed its own printed notices. The Foundation was means of identifying projects that not selective in this distribution, with could best meet its goals. the intent of reaching a broad audience of prospective grantees. From its inception, the Program has relied upon both external peer review and internal staff evaluations Initial Program funding was Number of Eastern (Blue) and to make funding recommendations. directed at research projects Western (Brown) Proposals The significance of each of those conducted in eastern forests Receiving Initial Year of Support review processes, however, evolved because of greater under- over time. Initially, research scien- standing of forest bird tists with well established records issues in those areas. That and projects implemented in the trend disappeared, however, eastern United States were favored by the Program’s third year. because of the need to help ensure the success of this new program, and because of the greater availabil-

26 OBJECTIVE 2: SUPPORT QUALITY PROJECTS ity of background information on scientists, managers, and conserva- Total Annual Funding eastern forest bird issues, respec- tionists sensed the possibility of a Provided by the tively. That bias disappeared quickly, significant increase in available Foundation for Eleven however, as more interest was funding. While that did occur at Focal Research Projects generated from the West. many levels, the bulk of additional funds were directed at new positions Program staff initially shouldered the and activities other than research. bulk of proposal evaluation. But, by 1993, the level of external peer Foundation funding for forest bird review was enhanced. For each research was timely but, despite the proposal, Program staff attempted gradual increase in Program to obtain at least two independent contributions to research (reflected 1991 1999 assessments from government in these 11 projects), the extent of agencies, nongovernmental organi- funding was never enough to hold zations, or industry biologists the attention of many research familiar with forest management biologists who originally attended issues and bird ecology. Using both Partners in Flight meetings. In time, internal and external reviews, a the participation of research biolo- more inclusive Foundation staff met gists in Partners in Flight waned. three times per year to discuss all Foundation proposals, including those on forest management. Evaluation of Objective 2 Program evaluators examined each of the 37 proposals funded as part Provide Support for Quality Projects of the 11 projects for strong experi- mental design and methodologies, Given the level of funding, the Foundation made a justifiable timetables, and realistic reasonable effort to inform prospective scientists of expectations for addressing priority the availability of funding for forest-based research on information needs. For 35 of the 37 birds. Most of those efforts were directed at research- proposals, projects were well ers who were active participants in Partners in Flight, developed and justified and offered reasoning that those individuals would be most cogni- reasonable expectations for project zant of the issues deemed critical to success of the objectives. The other two proposals had significant merit, but principal initiative. In most years, requests for funding far investigators overstated the extent exceeded that available. In addition, the Program has of work that could be accomplished been able to attract and support some of the premier in the stated timeframe. Neverthe- researchers in forest bird ecology. Additional Founda- less, that the vast majority of the tion funds directed at bird research would re-establish proposals funded by the Foundation the important role that research has in conservation were of extremely high quality initiatives, such as Partners in Flight. reflects well upon the review process established by Program staff. Given the level of staffing, the Program performs exceptionally well in securing external peer and inter- Availability of Funds nal staff reviews for proposals submitted for funding. An independent assessment of the quality of the 37 Prior to 1990, few funds were research proposals funded in this evaluation found available to biologists outside of that nearly all aspects of all projects were of the high- natural resources agencies to est quality and were deserving of support. initiate research on management of EXCELLENT non-hunted forest birds, and even those resources were meager. With the advent of Partners in Flight,

27 EVALUATION OF FOREST RESEARCH

OBJECTIVE 3 STIMULATE ADDITIONAL RESEARCH

Stimulate additional avenues of research quality research project is Two projects were especially note- and areas of inquiry Acharacterized, in part, by its worthy in this area. In Minnesota, on forest bird ability to stimulate additional lines of collaboration between the Depart- management and inquiry. The 11 projects supported ment of Natural Resources (DNR) conservation by the Foundation were each and the Natural Resources Re- examined for evidence of a “spring- search Institute advanced from board” effect (see sidebar to left) documentation of bird-habitat that would lend support to the relationships and establishment of presumption that these projects region-wide monitoring, to develop- served in a synergistic capacity and ment of sophisticated mathematical influenced the advancement of models aimed at projecting long- science well beyond the systems in term impacts of various land uses which the original work took place. on bird populations.

Manomet’s work in Maine’s indus- Evolution of Research trial timberlands progressed from assessment of bird-habitat relation- A common theme among the ships, to development of heuristic projects was the evolution in direc- models that examined the tradeoff tion of research as answers to between economic and conserva- original questions became more tion benefits, to inclusion of multiple transparent. For example, original taxonomic groups of organisms in work in Alberta’s aspen forests assessment of the overall health of examined the response by bird forest ecosystems. populations to the effects of frag- mentation. Once preliminary infor- A program that supports continuity mation on that issue was secured, of both research teams and lines of emphasis was placed on under- research can be a cost-effective and standing movement of birds through efficient means of addressing those fragmented landscapes and complex ecological issues (e.g., its implications for the spatial design NSF 2002), given that the original of timber harvesting practices. areas of inquiry were of importance (Objective 1) and that projects and Research projects in Idaho reflected investigators were of high quality a shift in emphasis from stand-level (Objective 2). Projects sponsored by A springboard effect is studies in early years to more the Program were encouraged to exhibited when the results thorough examination of landscape- evolve over time, changing research of a project lead to investi- and regional-level dynamics in latter emphasis to accommodate knowl- gation of additional research years. Research in the Arkansas edge gained in prior years. In turn, questions and produce Ozarks initially examined habitat this allowed most projects to pro- results that, when inte- use and reproductive performance gressively investigate more complex grated, offer a more com- of birds in undisturbed forests, but issues, which were likely more prehensive understanding later used that study for comparison reflective of the ultimate information of larger ecological issues. with different silvicultural treatments. needs of land managers.

28 OBJECTIVE 3: STIMULATE ADDITIONAL RESEARCH

In addition to the natural evolution of Students working on research questions, projects sup- Foundation-supported ported through the Program were MOFEP research present notably effective in stimulating results of their indepen- research not directly or originally dent investigation at the intended to be funded by the 14th Annual Undergradu- Foundation. These efforts originated ate Research Science both within and outside of the core Symposium at the research teams. University of Missouri. More than 200 students have developed research Springboard Effect: Internal projects through MOFEP.

Internal efforts often were in the form Original bird and habitat surveys in of graduate and undergraduate Minnesota that were funded through student projects. The approach of the Program led principal investiga- the University of Missouri, in particu- tors there to expand into examina- lar, highlights the significant contri- tion of edge effects, riparian forest butions that these projects made to management, and silvicultural the advancement of science. In the practices. first six years of work, approximately 180 undergraduate and 12 gradu- Reproductive and habitat data from ate students worked on the bird birds in Arkansas forests were used portion of the Missouri Ozarks by the principal investigator in Forest Ecosystem Project developing into far-reaching theo- (MOFEP). All undergraduates were retical models of avian life history. required to conduct independent research projects. Projects were Manomet also exceeded expecta- diverse in nature, ranging from tions by building upon their experi- botanical aspects of silviculture ences in northern hardwoods forests practices to the effects of those and developing an objective, practices on small mammal popula- quantitative approach for prioritizing tions. Several of these projects led species for conservation action. to more in-depth investigation through graduate theses.

Attempts to develop decision support tools for land managers in Manomet Center for the Little Pee Dee region of South Conservation Sciences Carolina led researchers to create a developed a GIS-based spatially-explicit model (Habplan), process for identifying that allows foresters to better species at greatest risk schedule timber harvest in the in forested landscapes. context of environmental concerns, This is an example of such as wildlife habitat and sedi- the springboard effect, mentation of adjacent streams. whereby research teams pursue investiga- Aspen fragmentation studies in tion of issues not Alberta have led those scientists to considered in original further investigate types of predation research proposals. pressure on bird nests and how characteristics of landbird popula- tions vary with regional patterns of habitat quality.

29 EVALUATION OF FOREST RESEARCH

Springboard Effect: External McCallum 2003). Forest bird research led by the University of The ability of projects to stimulate Arkansas helped shape research additional lines of research by directions for Partners in Flight scientists outside of Program- (Donovan et al. 2002). supported research teams was assessed directly and indirectly in Stimulated by their research in this evaluation through three Idaho forests, principal investigators means: (1) direct evidence that of both projects in that state collabo- Program-supported projects pro- rated in development of a landmark vided the impetus for additional symposium and published proceed- research; (2) citation of supported ings, Predicting Species Occur- projects in research needs assess- rences: Issues of Accuracy and ments of agencies and organiza- Scale. This effort brought together more than 100 research scientists Researchers from Turn- tions; and (3) citation of supported to share information and chart stone Ecological Research projects in published reports. direction for future research in this Associates and Sustainable Though thorough investigation of field of study. Ecosystems Institute were this issue was not made by evalua- instrumental in bringing tors because of limited resources, Research conducted by Manomet together more than 100 information that was collected does Center for Conservation Sciences scientists to evaluate provide an indication of the level of contributed significantly to the current developments in impact of Program-supported direction of forest research in Maine landscape-level predictive projects on the scientific community. through its participation in the modeling and to identify The following represents examples Cooperative Forestry Research Unit avenues for future research. of ways in which Foundation sup- ported projects stimulated and at the University of Maine. In guided research beyond that funded addition, Manomet’s research had by the Foundation. substantial influence on forest research through the Forest Eco- Landscape-level research by NCASI system Information Exchange that it researchers in South Carolina established (e.g., Hagan et al. provided the foundation for develop- 2002), the purpose of which is “to ment of a broad study by North be an interface between those that Carolina State University (Thomp- can generate information and those son et al. 2002) examining repro- that need information about the ductive biology and habitat selection region's forests.” by Swainson’s Warblers, a high priority species in southeastern The stated research needs of other United States. agencies and organizations were influenced as well by research Research conducted by Avifauna supported through the Program. Northwest and Weyerhauser Superior National Forest in Minne- Company scientists on salvage sota identified priority research logging in Oregon stimulated more needs based, in part, on the re- in-depth research on sensitive search delivered through the species, such as the White-headed Minnesota DNR and Natural Re- Woodpecker. sources Research Institute. Facilita- tion efforts by the Tennessee The direction of research for Conservation League, through the Canada’s Sustainable Forest Southeastern Working Group of Management Network benefited Partners in Flight, helped govern- from studies conducted by the ment agencies, nongovernmental University of Alberta and supported groups, and private industry identify by the Program (Hannon and priority research information needs.

30 OBJECTIVE 3: STIMULATE ADDITIONAL RESEARCH

Numerous publications were generated from the Foundation- sponsored projects and, in turn, many of those works are regularly cited in other proposals and litera- ture, ostensibly stimulating addi- tional research. For example, publications resulting from forest research conducted at the Univer- sity of Alberta have been cited 163 times to-date in the published literature (source: Science Citation In 1999, Manomet Center Index). for Conservation Sciences created the Forest Ecosys- tem Information Exchange, a forum for public and private groups and citizens interested in sustainable management of Maine’s Evaluation of Objective 3 forests. Its mission, through open discussion of forest Stimulate Additional Research research, is to identify and fill information gaps in conservation and manage- The 11 projects supported by the Foundation stimu- ment of forest resources. lated a significant amount of research -- often well The Forest Ecosystem beyond that outlined in original proposals. In addition, Information Exchange was the supported projects helped chart the direction of one of the most effective other research efforts. means by which any of the eleven projects shared Supported principal investigators were encouraged by scientific information and Program staff to adapt subsequent research propos- stimulated additional als to scientific information collected during prior research. years. When appropriate, many research teams re- sponded to that charge, which allowed several studies to progressively investigate more complex -- and likely relevant -- forestry issues.

The springboard effect was clearly evident throughout this collection of projects. Most studies stimulated valuable additional research, both within and outside of the funded research team. The cumulative result was development of novel approaches to scientific investigation that addressed many of the critical needs of forest managers. Clearly, maintaining the continuity of research teams over multiple years was instrumental to achieving these successes.

EXCELLENT

31 EVALUATION OF FOREST RESEARCH

OBJECTIVE 4 GENERATE VALUABLE INFORMATION

Data and information derived from supported o maximize use for forest of the stated objective. Objectives research projects are Tconservation and management, not fulfilled reflected those rare of high value to land results from research projects occasions on which the planned managers, planners, needed to produce results and have work failed to deliver viable informa- and other outcomes reflective of three principal tion or the intended product. characteristics: (1) the stated decision-makers objectives and goals of each project The cumulative number of objec- were met; (2) results were definitive tives for each project ranged be- with respect to research hypotheses tween 2 and 14. Evaluators recog- and objectives being addressed; nized 4 to 6 unique objectives for and (3) results and products had most (8) of the eleven projects. direct relevance and clear applica- tion to management and conserva- An average of 75% of the stated tion of forest resources. Objective 4 objectives were fully fulfilled for each was evaluated based upon assess- project (range = 50-100%). Approxi- ment of these outcomes. mately 22% of objectives within a project were partially fulfilled. Only two projects had an objective that Achieving the Objectives was not fulfilled (see figure to left). While this measure of success is Original funding decisions by the subjective, and therefore open to Foundation were based, in part, on different interpretations, the results the likely contributions each project outlined here suggest that project could make to conservation based investigators consistently delivered upon the stated project objectives the information and products that and goals. For each of the eleven were originally proposed to, and Performance of Projects projects, Program evaluators funded by, the Foundation. in Achieving Stated summarized all stated objectives Objectives over all years of funding, and No relationship was detected combined any objectives that were between the number of objectives similar. Evaluators then determined pursued by a project and its suc- the extent to which each objective cess in meeting those objectives had been met (as of 2003), through (Pearson’s correlation, r = -0.15). No examination of final project reports obvious difference existed between or other published or unpublished this measure of success and the documents, and via discussions type of organization (public or with principal investigators or other private) to which Foundation funds Bars represent the average personnel familiar with the project. were directed. percentage of objectives for Performance for each objective was each of 11 projects that classified as either (1) fully fulfilled, Objectives related to development of were fully met, partially met, (2) partially fulfilled, or (3) not forest management recommenda- or not met at all. Overall, fulfilled. Partially fulfilled objectives tions often failed to be fully met. 75% of objectives were fully were those that lacked sufficient Seven of the 11 projects identified achieved. effort or data to truly meet the intent that type of objective, but only 4 of

32 OBJECTIVE 4: GENERATE VALUABLE INFORMATION those 7 (57%) projects fully fulfilled scientists and practitioners of that obligation (compared to 75% natural resources management that successful completion for all objec- collaborate closely on both the tives). Overall, only 4 of 11 (36%) research design and subsequent projects directly generated specific, management recommendations. well documented land management Several approaches used by recommendations. investigators in this evaluation, including those in Maine and Difficulties developing management Tennessee, could serve as models recommendations may be the result for future Program projects. Partnership facilitation of several factors. First, while most efforts by the Tennessee projects received several years of Conservation League (TCL) Foundation support, research Definitive Results were pivotal in development scientists often are reluctant to of the first truly effective propose management guidelines on All projects delivered high quality regional working group of complex issues that may not be information related to objectives and Partners in Flight. Lessons completely elucidated within the research hypotheses. This signifi- learned through these limited timeframes and spatial cant Program accomplishment is efforts, which stressed close contexts of their research projects reflected by the large number of communication and collabo- (e.g., Richter and Redford 1999). works published in respected ration by all stakeholders in scientific journals (see below and bird conservation, provides Second, academic scientists may assessment of Objective 5). The some of the groundwork for not be especially familiar with many quality of information produced is an future large-scale conserva- of the on-the-ground practices that outgrowth of rigorous study designs tion initiatives. define natural resources manage- (see Objective 2). ment (Meffe 1998) and, subse- quently, may not feel comfortable However, much of the information interpreting scientific data in the collected has yet to be published. To context of management actions. evaluate the ability of those data not yet published to deliver reliable And finally, several research projects information (sensu Romesburg funded by the Foundation did not 1981), Program evaluators exam- focus on specific issues of direct ined characteristics of data sets management concern. Hence, the included in final reports. Assess- breadth of the issue being investi- ment was made regarding (1) gated in some cases may not have concordance between the spatial allowed easy translation to manage- scale of investigation identified in ment recommendations. proposals and that represented by the data, and (2) minimum sample At least two actions could help size necessary to effectively address remedy the limitations of most the ecological issue of interest. The research projects (not only those in latter query was based upon attain- this evaluation) in developing ment of a minimum sample size relevant management recommenda- suggested for different types of tions. First, scientists and funding analyses. Specifically, sample sizes agencies should strive to develop necessary for multivariate analyses long-term projects whenever pos- (n > 10 times the number of predic- sible to help reduce the uncertainty tor variables; Stevens 2001) and that is inherent to short-term estimation of nesting success (nests environmental research investiga- per species > 20 per treatment per tions (Hilborn 1987). And second, year; Martin et al. 1997) were identification of sound management assessed. Both types of analyses recommendations could be acceler- were regularly used in projects. ated through creation of teams of

33 EVALUATION OF FOREST RESEARCH

Information included in annual However, several prospective users reports demonstrated that data were found that the information was not collected at spatial scales compat- directly beneficial to land managers. ible with specific ecological ques- Two reasons for this were cited. tions under investigation. For First, in some cases, the research example, regional-level issues information was not presented in a usually examined questions from format that allowed ready assimila- study sites positioned over large tion by non-scientists. This issue -- areas -- millions of hectares. Like- effective communication -- is more wise, data were collected over tens aptly discussed under Objective 5. The Red-eyed Vireo is a of thousands of hectares for land- good example of a species scape-based questions. These The other concern was that some that is relative easy to observations indicate that investiga- results required additional synthe- survey because it is vocal, tors closely adhered to study design sis, or steps, before data had direct but for which nest monitor- protocols outlined in proposals. relevance to management actions. ing can be fairly difficult In some cases, concurrent examina- (because of nest crypticity). Nearly all studies that examined tion of numerous species made All projects collected sound habitat use had adequate sample management implications nebulous survey information, but sizes (numbers of point counts and because of varied responses by many studies were less detections of various species) to different species to silvicultural successful in estimating develop rigorous predictive models. practices. In the absence of out- nesting success. Nest productivity data, on the other reach materials that synthesized hand, were marginal for 4 of the 5 and “distilled” these complex studies examined. This suggests patterns, immediate use by practi- that traditional survey approaches tioners of forest management was for studying avian responses to not always achieved. In addition, the silvicultural practices were well relevance of landscape-level phe- executed in these studies, whereas nomena was not always placed nontraditional methods, such as logically in the context of land nesting success, were much more management options. difficult to effectively implement.

Filling a Gap? Direct Application Traditionally, applied avian research Program evaluators assessed, directed at issues of high impor- primarily through interviews with tance to land managers has not prospective users, whether informa- been aggressively pursued by the tion resulting from each project research community. To what extent could be readily applied to issues of did the Program fill this lingering forest management. Virtually all need? respondents believed that the data had high potential to contribute to Overall, the Program was effective in forest conservation and manage- generating information that was of ment. Because of the often-stated great value to practitioners of forest “disconnect” between research management, planning, and policy. scientists and potential users of The majority of respondents recog- research information, this general nized the value of the scientific viewpoint by practitioners of forest information generated through the conservation and management research projects, and this senti- represents a notable success of the ment cut across the diverse group Program. of public and private organizations that serve as stewards of North American forests.

34 OBJECTIVE 4: GENERATE VALUABLE INFORMATION

Information generated by Manomet Center for Conservation Sciences...has very rapid application to our forests.

Si Balch Technical & Planning Superintendent MeadWestvaco Corporation

Evaluation of Objective 4

Generate Valuable Information

In general, the supported research projects delivered the information and products that were identified in the funding proposals. Deficiencies in that record, however, were evident for projects proposing to develop forest management recommendations; only 57% of those projects generated specific recommen- dations. In total, only about one-third of the 11 studies produced rigorous management recommendations.

All studies generated priority and defensible informa- tion. Research most often fell short, however, in studies involving the assessment of nesting success.

Most information was readily usable by forest manag- ers, although several criticisms were leveled at the utility of data that had not been synthesized ad- equately for practical use.

Overall, forest managers, conservationists, and policy makers were exceptionally pleased with the rigor and breadth of scientific information produced through the Program. That information offered numerous public and private organizations guidance for more effective forest management and conservation. Research conducted by the University of Arkansas...is profoundly useful.

GOOD Larry Hedrick Team Leader, Integrated Resources Ouachita National Forest

35 EVALUATION OF FOREST RESEARCH

OBJECTIVE 5 DISSEMINATE INFORMATION

Information from EFFECTIVELY supported research is effectively critical component of any sponsored project, but was funded communicated to Aresearch program is the assur- through and credited to another prospective users ance that information generated source. through projects is delivered in a timely fashion and in an appropriate Many internationally recognized format to target audiences. This journals served as outlets, including evaluation focused on the extent to the Journal of Wildlife Manage- which principal investigators and ment, Forest Science, Ecological other “spokesgroups” (collaborators, Applications, Forest Ecology and end users, and Program staff) Management, and Conservation communicated results of research Biology. In addition, several sympo- projects through written and verbal sium proceedings and agency means. Four principal outlets for this technical reports were used dis- information were considered in this seminate research results. These evaluation: (1) technical publica- publications were successful in tions such as peer-reviewed scien- bringing research results to the tific journal articles and book larger scientific community, the chapters; (2) outreach documents primary readers of those sorts of such as internal technical reports, publications. information circulars, and popular publications; (3) general presenta- In general, principal investigators tions at meetings of professional affiliated directly with state or federal organizations and special interest government (through a university) groups; and (4) targeted meetings were more effective in generating with prospective end users, includ- peer-reviewed publications than ing industry and conservation those associated with private organization staff. institutions. Overall, government-led studies produced an average of 8 peer-reviewed publications, com- Technical Publications pared to only about 3 publications for each of the private organizations. At least 62 scientific publications This discrepancy is likely related to were produced from the 11 projects, the incentives offered for publication an average of nearly 6 publications by those different types of institu- per study. Exact numbers of publi- tions. That is, academic promotion cations were difficult to assess of university-based scientists is because (1) scientists often had usually tied closely to the number several concurrent projects under- and quality of peer-reviewed publi- way and some publications drew cations. In addition, university upon more than one of those professors often generate publica- studies, and (2) some publications tions through students working on may have originated, or been graduate degrees through these stimulated, through a Program- projects.

36 OBJECTIVE 5: DISSEMINATE INFORMATION EFFECTIVELY

Private organizations, on the other Scientists from hand, usually do not have that same Manomet Center for level of organizational incentive to Conservation Sci- publish in peer-reviewed outlets. ences published the Rather, the overriding requirement results of novel of those investigators is often a final research aimed at report to an organization that better understanding provided funding. Furthermore, the effects of silvicul- whereas academic scientists usually tural practices, and have officially recognized time for the landscape-level development of scientific works, context of those their counterparts in private organi- activities, on abun- zations may not have that level of dance of forest- “luxury.” nesting birds in Maine. This work was published, in part, in Outreach Documents the journal, Forest Science. Outreach information in this evalua- tion includes documents that summarized the main results of a project, especially that information that could be used directly and readily for management or conser- vation purposes. The principal target audience was practitioners of forest management.

Fewer than one-third of the projects invested significant effort in develop- ing outreach materials. In defense of the pool of projects, outreach materials were rarely stated as an objective in proposals. So, despite its value, few projects were expected by the Program to develop this type of communications vehicle.

The outreach efforts of three projects warrant mention. A principal focus of the work of the Tennessee Conservation League was to develop partnerships, the outcome of which was, in part, outreach University of Missouri ecologists materials that could help guide were involved in one of the most implementation of science-based ambitious experimental examina- conservation by those partners. tions of silvicultural practices ever Numerous outreach documents undertaken in the United States. were produced through these More than a dozen scholarly works on bird ecology have efforts, including identification of been published from this study, including this article in priority habitat types, bird species, Conservation Biology. and management and conservation actions.

37 EVALUATION OF FOREST RESEARCH

The Program-supported work of sented information at meetings of Minnesota Department of Natural the forest products industry. In total, Resources ultimately led to creation at least 150 presentations (again, of two widely acclaimed outreach difficult to precisely measure) were publications (though not directly made to national and regional funded by the Foundation). The two scientific, conservation, and man- booklets were aimed at land stew- agement societies. ards concerned about maintaining the sustainability and ecological integrity of forest tracts. Targeted Meetings

Manomet Center for Conservation This type of information dissemina- Sciences created an outreach tion is intended to bring specific newsletter, Mosaic Science Notes, Outreach booklets devel- information directly to relatively that provides details to forest oped by the Minnesota small groups of people who can managers on the outcome of Department of Natural directly apply the information to research conducted on industrial Resources was an out- forest management and conserva- timberlands. Each issue of Mosaic growth of work funded by tion. Program evaluators assessed focuses on a single forest issue and the Foundation. Booklets this form of communication by provides data summaries and were intended to assist both examining the extent to which graphics, and outlines the implica- large (top) and small research teams provided specific tions the results may have for forest (bottom) landholders in information to prospective users management. Forest managers and managing their forests in a through one-on-one or small group conservationists in Maine praised sustainable and ecologically meetings with forest products this form of information dissemina- sound fashion. industry personnel, conservation tion. groups, or other land stewards.

All research teams disseminated General Presentations information to some extent by this method, although efforts ranged Scientific research supported by the widely from a single annual meeting Foundation was well represented in with collaborators to more regular meetings of national and interna- meetings to discuss research tional scientific societies, including progress and results. NCASI the Society for Conservation Biol- scientists, for example, regularly ogy, Ecological Society of America, met with individual groups to share Cooper Ornithological Society, and information. The Manomet research American Ornithologists Union. team interacted regularly with the State-level meetings of The Wildlife timber industry in Maine and has Society also were used regularly as held informational field trips to study a forum for these projects. areas for state and regional conser- vation groups. Their Forest Ecosys- Project results also were presented tem Information Exchange also at several symposia on forestry, offered an effective forum for discus- ecosystem-level management, and sion of research results. bird conservation. The efforts of the Tennessee Conservation League Tennessee Conservation League and several other research teams facilitated numerous meetings each supported through the Program year to share information. Others, were showcased at national and such as scientists from the (Minne- regional meetings of Partners in sota) Natural Resources Research Flight. Research scientists from the Institute and University of Alberta, National Council for Air and Stream shared scientific information Improvement (NCASI) often pre- through standing committees

38 OBJECTIVE 5: DISSEMINATE INFORMATION EFFECTIVELY involved with state or provincial Other Contributions initiatives on forest management. As expected, publication in peer- Small, targeted meetings between reviewed journals was not a regu- researchers and prospective users larly used outlet for project collabo- of scientific information probably rators, end users, or Program staff. was the weakest form of information However, several collaborators and dissemination examined in this end-users presented research evaluation. This was likely a result of results at meetings. For example, the fact that this type of activity Ed Lindquist, Forest Wildlife Biolo- (technical assistance) is not seen as gist at Superior National Forest a priority to many institutions within presented results of the Minnesota which some of these scientists work. research to prospective end users.

Mosaic Science Notes, developed by Manomet Evaluation of Objective 5 Center for Conservation Sciences, is intended to Disseminate Information Effectively provide a summary of scientific research that can be directly applied Presentation at large, national and regional meetings was the to forest management most consistent means by which supported projects presented and conservation. This research results. Overall, publication of results in peer-reviewed information has been outlets was significant across the pool of projects, but was subsequently presented inconsistently applied. In general, researchers directly affiliated by collaborators at with universities were more productive in publishing than meetings of prospective scientists working for nongovernmental organizations. The end users of the infor- combination of national-level presentations and publications mation. For example, Si were the principal mechanisms by which scientific information Balch of Mead was disseminated to prospective users. Westvaco regularly made presentations to industry and other Unfortunately, those two approaches are likely the least effec- groups about tive in delivering scientific information to local practitioners of Manomet’s research forest management and conservation. More effective are likely results and its applica- to be targeted outreach materials and small group meetings, tion to forest manage- both aimed at local and regional land managers. Unfortunately, ment. these avenues of information dissemination were irregularly taken by supported research teams.

Thus, this Program evaluation has identified a notable gap in the effective dissemination of research results. While peer- reviewed publications and presentations before scientific soci- eties are critical for judging the legitimacy of research results, those outlets probably are not highly effective in delivering information to practitioners of forest management in a form that are readily and immediately used.

GOOD

39 EVALUATION OF FOREST RESEARCH

OBJECTIVE 6 INCORPORATE RESULTS INTO PLANS

Information from supported research his objective evaluates the most began in the late 1980s (Morrissey projects provides Tfundamental level of commit- et al. 1994, Sustainable Forestry management and ment by forest management and Board 2002), about the same time conservation guidance conservation organizations to as the inauguration of the Program, to prospective users incorporate research information had a significant influence on ways into organizational policies and in which industry approached forest management plans. It does not management. Because of those consider actual implementation of concurrent efforts, the influence of forest management actions; that Program-specific outputs on forest issue is covered under Objective 7. management policy was sometimes The level of this commitment was difficult to discern from actions documented through examination of influenced by the larger, more the policies and plans of three types general movement. of organizations: (1) forest products industry, (2) government agencies, Program-sponsored research did and (3) conservation organizations. not have a broad impact on man- agement plans of the forest prod- ucts industry, although several Forest Products Industry significant outcomes did occur (see below). In many cases, scientific The movement towards better information generated from research stewardship of plant communities, projects did not reach the hands of animal populations, and other private forest managers or, if it did, “nontraditional” forest values that was not in a format that was consid- ered to be readily usable. Plum Creek Timber Com- pany in Maine adopted the The limited use of research informa- forest patch retention tion by industry did not appear to model developed through follow any prescribed pattern across the research of Manomet the suite of research projects. The Center for Conservation most notable example of industry Science. The retention use of information was documented model stresses the need to in Maine, where the research team retain small patches of from the Manomet Center for overstory trees within Conservation Sciences worked on landscapes dominated by private lands, regularly discussed even-aged harvesting research needs with industry practices, such as personnel, and made concerted clearcutting. Research attempts to understand the con- suggests that these small straints under which industry stands reduce the poten- operated. Those efforts resulted in tially negative impacts of significant impacts on the manage- clearcutting on bird ment philosophy, harvesting plans, populations inhabiting and interest in biological issues, adjacent uncut forests. exhibited by several corporations.

40 OBJECTIVE 6: INCORPORATE RESULTS INTO PLANS

Plum Creek Timber Company, for Mead Westvaco Corpora- example, adjusted forestry plans in tion worked with Ma- Maine based upon the forest patch nomet Center for Conser- retention models developed by vation Sciences in Maine Manomet (see sidebar to left). Other to develop guidelines for research-driven aspects of biological more effective protection diversity included in management of streamside manage- plans were nesting raptors and ment zones and the downed woody debris. plants, animals, and other life that depends Mead Westvaco Corporation also upon those unique used Manomet’s research informa- habitats. Manomet tion extensively in development of research examined the management plans, including affects of different buffer incorporation of biological diversity, strip widths on physical ratios of different habitat types, and and biological properties forest patch retention concepts. In of the streamside zones. addition, this large corporation created management guidelines for riparian zones in consultation with Manomet (see sidebar to right). assembly of forest managers, scientists, and land use planners. International Paper used information That information was considered in from several Foundation-funded development of management plans research projects to help shape its for several large, private landhold- silvicultural activities for the benefit ers. of wildlife and plant communities. Research in Arkansas, Oregon, NCASI held a unique position South Carolina, Idaho, and Alberta among Foundation-funded organi- included direct participation by zations because of its close affilia- industry biologists. Industry in those tion with the forest products indus- areas have used that research try. While the evaluation identified information to some extent in no substantive changes to industry developing management plans. management plans that could be attributed directly to NCASI-led Use of research information by research, industry approaches to timber-based industry was likely a forest management almost certainly function of the extent to which were influenced through regular research teams directly and regu- interactions between NCASI scien- larly engaged industry personnel. tists and industry colleagues. The influence of Manomet research results on the actions of the pri- Tennessee Conservation League vately-owned land base in Maine, for engaged both small and large example, clearly was a result of a private landholders within the priori discussions between scientists Interior Low Plateau ecoregion of and land managers, active coopera- Kentucky, Tennessee, and Alabama, tion of those land managers in including Champion International, implementation of research, and Westvaco Corporation, and creation of a targeted discussion Willamette Industries Inc. Land forum that included an open invita- management recommendations tion to interested stakeholders. intended to maintain the viability of While the evaluation did not docu- forest bird populations were collabo- ment a broad influence on manage- ratively developed by a diverse ment plans and policies of industry

41 EVALUATION OF FOREST RESEARCH

Information for Statewide and other private landholders by federal lands in Tennessee. Missouri Forest Sustainability Program-sponsored research, Department of Conservation intends several industry representatives to use results of the University of Most state governments indicated that the scientific informa- Missouri bird research to shape have responsibility to tion did help initiate and continue future policy on timber harvest. ensure the long-term discussion of novel approaches to sustainability of forest sustainable forestry. Results of research conducted by resources. This is not a Manomet Center for Conservation simple task given the Sciences had substantial influence competing demands on Government Agencies on revision of existing regulations forests. Don Mansius, within the state legislature. Scien- Director of Forest Policy and Research information generated tists with Manomet conducted a Management for the Maine through the Program did not have mathematical simulation of the Forest Service offered his far-reaching impacts on public land projected outcome of the Maine thoughts on what Founda- management, but did affect numer- Forest Practices Act and deter- tion-sponsored research ous local policies and management mined that the existing policy would has meant to his agency plans of agencies charged with not deliver the projected benefits and to sustainability of overseeing forest lands. (Hagan and Boone 1996). In turn, Maine’s forests. the state legislature, based largely The USDA Forest Service was on that research, adopted new „ Provided state-of-the-art, perhaps the greatest beneficiary of legislation that was considered to be cutting-edge information research information. Conclusions more realistic with respect to forest that is easily digestible drawn about the effects of silvicul- sustainability. In addition, and understandable and tural practices on birds by the Manomet’s research results have is used by agency to University of Arkansas were used in helped shape the approach to forest establish regulations revision of the Ouachita National management and conservation Forest Management Plan in Arkan- developed by the Maine Forest „ Manomet provided sas and Oklahoma. Several national Service (see sidebar to left). information that is real, forests in Oregon incorporated timely, and they get it salvage logging recommendations into the hands of the from Avifauna Northwest into Conservation Organizations people who are actually recommended management doing the work practices. Superior National Forest Nonprofit conservation groups have in Minnesota and Chequamegon benefited from information gener- „ Value of research and Nicolet national forests in ated through the Program, but information was reflected Wisconsin used bird-habitat rela- probably less so than either govern- in the high marks tionship information produced by ment or industry groups. industry received on the Minnesota Department of annual certifications Natural Resources and Natural The Tennessee Conservation Resources Research Institute to League’s partnership-building craft more viable forest manage- efforts, which were emphasized ment plans. much more than research in their project, provided a forum through Provincial government biologists which conservation groups in accessed data from the University of southeastern United States could Alberta’s research on birds in access state-of-the-art information industrial timberlands to develop on bird conservation. policy for harvesting procedures in Canada. Both the U.S. Fish and Forest research in Arkansas and Wildlife Service and Tennessee Missouri has benefited, or is pro- Valley Authority sought guidance jected to offer future benefits, to The from the Tennessee Conservation Nature Conservancy and Audubon League and its partners for develop- Missouri, respectively. The Arkansas ment of management plans on Chapter of the Nature Conservancy,

42 OBJECTIVE 6: INCORPORATE RESULTS INTO PLANS

for example, used the bird research activities are being conducted with Northern Waterthrush to help justify its position on land the well-being of migratory birds in acquisition in the region. The mind than ever before. Northern Forest Alliance, Appala- chian Mountain Club, and other Second, both principal investigators groups believed that information and other scientists and forest generated by Manomet will help managers who are exposed to clarify the issues surrounding forest Foundation-supported research management in Maine (see sidebar results incorporate that information to right). into their working knowledge of forest management. Subsequent professional opportunities to have Intangible Benefits input into forest management practices may therefore be influ- An NGO Perspective Through interviews with prospective enced by exposure to those original research results. For example, one users of research information, three While research information principal investigator of Foundation- additional benefits to conservation was used by all prospective supported research, Dr. T.B. Wigley, were identified, all of which were user groups examined in used research results to guide his difficult to precisely quantify given this evaluation, nongovern- requested technical review of a the nature of the original assess- mental organizations may landmark regional forest assess- ment design. have been the least en- ment (see panel below). gaged in use of research First, most prospective users, results. Jonathan Carter, And finally, research results from particularly those in the forest Director of the Forest projects examined here helped products industry, indicated that Ecology Network in Maine, validate, and in some cases refute, Foundation-sponsored (and other) identified what he believed forest management principles and research projects increased organi- to be the fundamental prescriptions that had been widely zational awareness of the plight of benefits provided thus far by implemented in both public and migratory birds. Prior to 1990, few researchers from Manomet private forest tracts throughout forest managers incorporated the Center for Conservation North America. So, while the needs of nongame migratory birds Sciences. into forest management plans. research results may not have Through Partners in Flight and its changed existing management „ Brought science to the plans, results did in some cases associated programs, knowledge of forefront of decision- provide legitimacy to ongoing forest migratory birds has been pushed to making the forefront of forest management. management actions. The result is that more management „ Created a dialogue amongst groups with various viewpoints

„ Generated sound scientific information for forest planning and management (though data can be subject to different interpretations) Research results from the Program not only contrib- uted directly to scientific publications, but also helped shape the professional opinions of scientists who subsequently offer their informed viewpoints on other conservation- and management-based documents, such as this landmark effort to evaluate the ecological state and management needs of southern forests.

43 EVALUATION OF FOREST RESEARCH

What Limits Incorporation extensive use of research informa- of Research Information? tion? The general sense from this evaluation is that research results Scientific information produced often are not delivered to prospec- through the Program clearly was tive users in a form that can be used to develop organizational readily applied to forest manage- policies, practices, and procedures ment and conservation. So, despite of government agencies, nongov- the high quality and relevance of ernmental organizations, and the research being conducted through paper products industry. However, the Program, most researchers put use of that information was not minimal effort into outreach materi- extensive and usually limited to als directed specifically at practitio- properties on which research was ners of forest management and conducted. Why was there not more conservation (see Objective 5).

Evaluation of Objective 6

Incorporate Results into Plans

Organizational acceptance of scientific information is first reflected by its incorporation into official policies and manage- ment plans. Results from research projects supported by the Program were used in this way by numerous government agencies, private landholders, and conservation organizations. Hence, the research has potential to influence the actions necessary to maintain the viability of forest bird populations. However, two shortcomings in this outcome were identified.

First, while information from all projects was used to some extent by practitioners of forest management and conservation, only slightly more than half of the 11 projects were highly successful in having specific research-derived recommenda- tions incorporated into forest management plans. Investigators who were most successful had developed a close professional relationship with prospective users of the information.

Second, use of research information often was limited to stew- ards of the property on which the research was conducted. Research information directed at land managers was not dis- seminated widely.

Incorporation of scientific research results into plans for forest management is encouraging. However, more concerted efforts to engage forest managers and conservation groups could substantially enhance the use of that information.

GOOD

44 OBJECTIVE 6: INCORPORATE RESULTS INTO PLANS

45 EVALUATION OF FOREST RESEARCH

OBJECTIVE 7 IMPLEMENT MANAGEMENT ACTIONS

Recommendations derived from enefits of using research to Ouachita National Forest of Arkan- supported research Binform management and sas and Oklahoma were changed to projects are conservation actions ultimately reflect edge and area effects identi- implemented by depend upon timely implementation fied by the University of Arkansas. of recommended actions. Objective private, 7 addresses this critical step in the In Maine, both Plum Creek Timber nongovernment, and overall process of generating Company and Mead Westvaco government groups reliable information for the benefit of Corporation adjusted harvesting natural resources conservation. practices, including consideration of Land management organizations forest age-class distribution across were queried about the extent to the landscape, maintenance of which Program-sponsored research streamside buffer zones, inclusion had been incorporated into man- of retention patches within water- Total turnaround in bird agement actions. sheds, and identification of critical management on Superior habitat for sensitive species. As National Forest were the Though some attrition did occur, stated previously under Objective 6, words of Ed Lindquist, research-supported management research also provided validation of Supervisory Forest Wildlife actions identified in forest plans and several existing forest management Biologist, in describing the policy (see Objective 6) were practices being implemented by the impact of Foundation- implemented by land management forest products industry. sponsored research for organizations. Those types of forest conservation in actions, however, were sporadic and Tennessee National Wildlife Refuge northern Minnesota. Re- inconsistent across the suite of 11 adjusted management actions to search led by the Minnesota projects included in this evaluation. help promote Cerulean Warbler and Department of Natural other species habitat along the Resources and the Natural Tennessee River. Land manage- Resources Research ment actions of numerous other Institute generated statisti- Success Stories landowners in Tennessee were also cal models that predict the influenced by the management likely consequences of Nevertheless, several notable recommendations offered by the various forest management accomplishments were docu- Tennessee Conservation League. scenarios. Possibly most mented. Recommended size of significant in those models harvested forest patches in the was the capacity to predict the long-term impact of single timber sales on bird populations within and surrounding the affected stand. A prime consider- ation for evaluating the effects of harvest was the proportion of landscapes in various stages of forest succession, such as young forest pictured at right.

46 OBJECTIVE 7: IMPLEMENT MANAGEMENT ACTIONS

Significant management changes Tennessee Conservation were instituted on Minnesota’s League and its partners Superior National Forest as a result developed lists of high of the Program-sponsored research priority species, such as the (see sidebar to left). Snag retention Prothonotary Warbler (left), practices implemented on national that would allow both small forest lands in Oregon were influ- and large landowners to enced by the results of research manage their forests for the conducted by Avifauna Northwest. greatest benefit to wildlife.

Obstacles to Implementation

Lack of widespread implementation of management actions based upon Foundation-sponsored research was a reflection of the cumulative shortcomings identified in the assessments of the first six objec- tives. Specifically, the lack of clearly stated management recommenda- tions in several projects, along with ineffective dissemination of informa- tion to managers in other projects, created a bottleneck in the flow of Evaluation of Objective 7 useful information to institutions that have the potential to affect changes Implement Management Actions within forested landscapes. Several significant local changes in forest manage- Hindering implementation of man- ment have resulted from the research supported by agement actions, too, is the inher- the Foundation. On regional and national scales, ent time lag between presentation of new scientific information and however, that research has not yet had a widespread crafting of management plans that impact on the ways in which forest lands are man- incorporate that information. In aged. addition, several long-term projects were still underway in one form or The lack of consistency in local application of re- another, even several years after the search results, and the minimal influence those Foundation last supported the results have had thus far on land stewards outside research. Those projects are likely to the immediate study areas may be a consequence of bear additional future benefits to the the interplay among several factors, including: poor land management community. dissemination of management-based recommenda- Finally, incumbent upon managers tions; the time lag for collective research information is the responsibility to seek and (knowledge) to build to a “critical mass” often neces- assimilate information from a variety sary to stimulate action; and the lack of incentive for of sources, including the scientific many land managers to seek and assimilate the literature. Larry Hedrick, Integrated latest research information about wildlife conserva- Resources Team Leader, Ouachita tion and management. National Forest, stated... FAIR I don’t see how anyone can stay abreast of this job without staying abreast of the literature.

47 EVALUATION OF FOREST RESEARCH

OBJECTIVE 8 HABITAT GAINS

Habitat conditions are improved for onservation actions typically are Documented Habitat Gains forest-dwelling birds Caimed at improving the habitat conditions for a species of interest While management changes did and only indirectly at the species result from new research information itself. This is especially true for (Objective 7), land stewards could migratory birds that annually range not definitively document that those over thousands of square miles management changes improved across several countries. Habitat habitat conditions for forest birds. enhancement usually represents the The shortcoming here was not that most direct means of ensuring the those beneficial habitat changes did long-term viability of bird popula- not occur, but rather that land tions (Sherry and Holmes 1995). managers had not made an attempt to quantify the resultant changes in The evaluation recognized this the land base or natural resources. objective -- the improvement of forest habitat conditions for birds -- as the most direct and defensible Inferred Habitat Gains measure of the extent to which the Program benefitted populations of In light of the lack of documentation wild birds. Evaluators conceded about habitat improvements for beforehand the difficulty in reliably forest-dwelling birds, another quantifying habitat improvements. approach was used to infer the habitat gains that may have resulted The objective was directly assessed from Program-sponsored research. by examining the qualitative and For this evaluation, changes that quantitative evidence provided by occurred in management action land stewards that information were extended to the land base on derived from the Program stimu- which those changes were imple- lated beneficial changes in land mented to project how habitats management, planning, or acquisi- might have been improved. While tion and that those changes re- subjective, these projections are sulted in improve habitat conditions. based upon well documented Evidence for improved conditions Loss of understory conifer- patterns of habitat preference by would include, for example, reduced ous trees, such as spruce various guilds of forest birds. edge-to-interior ratios of forest and fir, are a concern to patches or riparian buffer strips, forest managers in Minne- On the Superior National Forest, increased coverage of forest types sota not only because of the snags (standing dead trees) are or age classes critical for high ramifications for long-term now not cut unless a safety hazard. priority species, or enhancement of sustainability of forest Snags are critical for both primary microhabitat conditions (e.g., forest resources, but also because (e.g., woodpeckers) and secondary understory vegetation) required by numerous species of (e.g., chickadees, Great Crested those species (e.g., Diamond neotropical migratory birds Flycatcher) cavity nesters (Evans 1975). Ideally, those improvements depend upon those tree and Conner 1979). Retention of should be measured in terms of species for nesting habitat. snags in Minnesota, Oregon, and acreage.

48 OBJECTIVE 8: HABITAT GAINS

Maine likely improved millions of acres of habitat for numerous cavity- nesting species. Snag retention practices Changes in the sizes, shapes, and were implemented in configuration of timber harvests several states after recom- probably resulted in reduced forest mendations were generated edge-to-interior ratios across through Foundation- forested landscapes in Alberta, sponsored research. Cavity- Maine, and Arkansas, improving nesting species, such as breeding habitat for species that are this 3-month-old Great sensitive to edges or patch size. Crested Flycatcher, stand to benefit from this enhance- For species that use landscape- ment in land management. level cues in selecting habitat, retention of small patches of over- story trees within clearcut stands may have improved the overall quality of forested landscapes.

Finally, several landholders have implemented significant shifts in their approach to land manage- ment, from consideration of only single species to incorporation of the habitat needs of multiple spe- Evaluation of Objective 8 cies. The likely result has been more effective conservation of ecosystems rather than simply on individual, Habitat Gains high profile species. This philosophi- cal change is likely to lead to greater Though previous objectives have documented sig- success in meeting the long-term nificant changes in forest management and policy, habitat needs of all species. no concerted effort was made by landholders to assess the extent to which those changes have potentially benefited forest birds through improved Tracking Habitat Changes habitat conditions.

A fair number of land management Simple inference based upon those implemented changes resulted from scientific management changes, however, suggests that information provided by Foundation- millions of acres of forest land in North America have supported research projects. Unfortunately, quantitative docu- benefited from scientific information generated by mentation of on-the-ground gains in Foundation-sponsored research projects. Neverthe- habitat quality has not taken place. less, reliable assessment of this objective cannot be The reason for this is understand- made given the lack of sound information about the able, given the staff time necessary extent of land use changes. to assess and monitoring the fine- scale metrics (e.g., edge-to-interior Long-term strategies designed to balance the mul- ratios) that might reflect changes in tiple goals of forest management need to incorporate habitat quality for bird populations. a fine-scaled assessment and monitoring protocol to However, use of monitoring to track changes in presumed habitat quality for forest assess management actions is necessary to ensure the efficacy of bird (and other wildlife) populations. those novel management actions.

49 EVALUATION OF FOREST RESEARCH

OBJECTIVE 9 BIRD POPULATION GAINS

Population status of forest-dwelling birds The ultimate goal of the Program is consistent source of bird population is improved to benefit populations of forest- data over regional spatial scales. dwelling birds. Relating bird popula- However, the number of routes for tion changes directly to manage- those surveys were not numerous ment actions, however, are ex- enough to generate legitimate tremely difficult to ascertain because comparisons of bird populations of various environmental factors act between areas that had manage- simultaneously on bird populations, ment changes implemented and such as breeding habitat limitations, adjacent areas where those winter food shortages, mortality changes were not made. during migration, and stochastic events such as severe weather. Nevertheless, given its importance, The Need for Population assessment of population change Assessment was necessary to include in the evaluation. The evaluation did not reveal any long-term assessment and monitor- This objective was evaluated directly ing programs aimed at understand- through interviews with land man- ing the impacts of forest manage- agement groups that had imple- ment changes on bird populations. mented actions based upon recom- While anecdotal information does mendations of research projects exist in several areas for assessing funded by the Foundation. the relationship between manage- ment and bird populations, those data are not substantial enough to Documentation of Gains drawn definitive conclusions.

Similar to that found in the assess- Population assessment and moni- ment of habitat gains, land manag- toring for animal and plant popula- ers were uncertain of the extent to tions must become a standard part which habitat management changes of forest management if that man- had affected local bird populations. agement is to evolve into a true In fact, because of the confounding multifaceted program. effects of the various environmental factors discussed earlier, land managers were even less specula- tive than they had been about potential habitat gains.

Few data exist to independently assess the extent of population changes. The Breeding Bird Survey of the U.S. Geological Survey is perhaps the most standardized and

50 OBJECTIVE 9: BIRD POPULATION GAINS

Evaluation of Objective 9

Bird Population Gains

Not surprising in this evaluation was the lack of qualitative or quantitative information about the effects of implemented forest management practices on bird populations. While positive effects are likely Forest management to have occurred in some areas that implemented practices developed with new forest management practices, reliable assess- input from researchers ment of this objective cannot be made given the lack supported by the Founda- of sound information about bird population tion probably had positive changes. influences on populations of forest-nesting birds, such Long-term strategies designed to balance the mul- as Swainson’s Thrush (top) tiple goals of forest management need to incorporate which prefer older-growth an assessment and monitoring protocol to track coniferous forests, and changes in bird populations inhabiting forested Common Yellowthroat (bottom), an inhabitant of landscapes. Without those measures, forest manag- very early successional ers may have little basis for implementing forest forests. However, land management practices under the banner of sound management agencies and conservation. industries have not insti- tuted assessment and monitoring protocols to evaluate the long-term effects of those practices on bird populations.

51 EVALUATION OF FOREST RESEARCH

The National Fish and CONCLUSIONS & RECOMMENDATIONS Wildlife Foundation should continue, and expand upon, verall, the forest bird research „ Strongly emphasize collaborative its commitment to Oprogram of the Foundation projects between researchers forest bird conservation produced significant and wide- and land managers to ensure through generation spread benefits for bird conservation that research is focused on high of quality scientific in North America. No other granting priority information needs and is program in the United States information. more likely to be used in develop- duplicates the intent and approach ment of management plans. of the Foundation. Because of that effectiveness and the unique niche „ Demand that research teams that it claims, the National Fish and effectively communicate manage- Wildlife Foundation should con- ment recommendations and tinue, and expand upon, its commit- research implications to forest ment to forest bird conservation managers. through generation of quality scientific information. Relatively few other substantive sources of funding „ In addition to peer-reviewed are available for applied research on technical reports, stress develop- forest birds and other wildlife. ment of research reports and Indeed, few other Foundations or outreach materials that target organizations that support research management-based end users are so closely tied into the informa- of the information. tion needs of forest managers. „ The Foundation should draw upon its supported research A Significant Concern projects and others to organize regular regional and national Of greatest concern was the lacklus- workshops on state-of-the-art ter communication of research forest management. results to land managers, and in a form that maximized its usefulness „ The Foundation should develop a to those end users. Mediocre “dynamic” forest management communication and dissemination manual that brings state-of-the- of results was likely a prime reason art scientific information to bear that research information was not on forest management in support more widely integrated into manage- of bird (and wildlife) conservation. ment plans, and implemented through stand- and landscape-level forest prescriptions. Other Recommendations

To overcome that major shortcom- The following table provides a ings, the Foundation should re- summary of all conclusions and emphasize several existing facets of recommendations drawn from this its Program, as well as add several evaluation. new components, including:

52 CONCLUSIONS & RECOMMENDATIONS

CONCLUSIONS RECOMMENDATIONS

IDENTIFY INFORMATION NEEDS

„ Visionary process that stressed „ Maintain active participation in participation of diverse stakehold- established working groups and ers, development of stable other discussion forums. process for identifying needs, and focus on restricted set of „ Develop, and solicit guidance management issues. from a panel of experts.

SUPPORT QUALITY PROJECTS

„ Identified proposals that were „ Re-emphasize importance of focused on priority management research information to forest issues; based upon partnerships management through mainte- with agencies and forest products nance and expansion of funding industry; led by respected opportunities. scientists; and underpinned by sound experimental design.

STIMULATE ADDITIONAL RESEARCH

„ Encouraged research teams to „ Continue to encourage research pursue complex, multifaceted, teams to pursue both basic and long-term research. Cumulative applied forms of research. result was development of novel approaches to scientific investiga- „ Maintain multi-year approach to tion that addressed critical needs funding of research projects. of forest managers.

GENERATE VALUABLE INFORMATION

„ Projects consistently generated „ Continue to stress to research high quality information that teams the need to produce high addressed priority needs of forest quality scientific information. managers. „ Demand, when appropriate, that „ Several shortcomings in produc- research teams develop rigorous tion of rigorous management management recommendations. recommendations.

„ Aspects of studies involving assessment of reproductive success sometimes fell short.

53 EVALUATION OF FOREST RESEARCH

CONCLUSIONS RECOMMENDATIONS

DISSEMINATE INFORMATION EFFECTIVELY

„ Presentation at national scientific „ Continue to stress the overriding meetings was the most consis- importance of peer-reviewed tent means by which project publications as a means of results were communicated. legitimizing and communicating research results. Ensure that „ The extent and quality of peer- funding is adequate to support reviewed technical publications scientists during publication were impressive, but was incon- development stage. sistent across the suite of re- search teams; researchers „ Demand that research teams affiliated with universities gener- effectively communicate manage- ally were more productive than ment recommendations to land those working for nongovernmen- management groups. tal organizations. „ In addition to peer-reviewed „ Presentation of research results technical reports, stress develop- and management recommenda- ment of research reports and tions were rarely developed outreach materials that target specifically for land managers. management-based end users. Outreach materials for both land managers and non-technical „ The Foundation should draw audiences (e.g., county planning upon its supported research commissions) were produced by projects and others to organize only a handful of researchers. regular regional and national workshops on state-of-the-art forest management. Research on the effects of silvicultural practices on „ The Foundation should develop a nesting migratory birds, “dynamic” forest management such as the Worm-eating manual that brings state-of-the- Warbler (right), has directly promoted bird conservation in the Missouri, Arkansas, and Oklahoma Ozarks region. In Missouri, informa- tion about the abundance INCORPORATE RESULTS INTO PLANS and distribution of Cerulean Warblers, for example, was „ Several significant outcomes „ Recommendations offered under used by the Missouri were documented for incorpora- previous objectives could help Department of Conservation tion of research results into better engage land managers in conservation activities. policies and plans of agencies, and deliver information in a more Data from the University of private landholders, and conser- useful format to this important Arkansas assisted the vation organizations. group of end users. Arkansas Chapter of the Nature Conservancy in „ Results from only half of the advocating land protection projects were directly used by strategies in the mountain- land managers. Use of research ous region of eastern information often was limited to Oklahoma. stewards of the property on which the research was conducted.

54 CONCLUSIONS & RECOMMENDATIONS

CONCLUSIONS RECOMMENDATIONS

IMPLEMENT MANAGEMENT ACTIONS

„ Several significant local changes „ Recommendations offered under in forest management have previous objectives could help resulted from research supported better engage land managers The success of Manomet by the Foundation. That re- and deliver information in a more Center for Conservation search, however, has not yet had useful format to this important Sciences in securing a widespread regional or national group of end users. commitments for inclusion impact on the way in which forest of research results into lands are managed. organizational policy and management „ The lack of consistency in local plans was application of research results, largely due to and the modest influence those the group’s results have had thus far on land effectiveness in stewards outside immediate communicating study areas are consequences of scientific passive dissemination of man- information to agement recommendations; time stakeholder lag for collective research infor- groups. In mation to build to a “critical mass” 2003, for necessary to stimulate action; example, and lack of incentive for land Manomet managers to seek and assimilate scientists the latest research information. sponsored an all-day field trip to industrial timberlands (above) for conservation HABITAT AND BIRD POPULATION GAINS groups from around New England to discuss forest „ Landholders made no concerted „ The Foundation should engage research and conservation. effort to assess the extent to forest managers in a discussion which changes in land manage- of the need for adaptive manage- ment have potentially benefited ment in assessing the outcomes forest birds. of new approaches to timber harvest or other manipulations of „ Simple inference based upon forest resources. Through those implemented management measurement of habitat and changes, however, suggests that landscape features, as well as millions of acres of forest land in bird populations, the potential North America have benefited benefits to birds and other natural from scientific information gener- resources can be estimated on ated by Foundation-sponsored large spatial scales. research projects.

„ No conclusions can be drawn with regard to the extent of changes in local or regional bird populations due to scientific information offered through Foundation-sponsored research.

55 EVALUATION OF FOREST RESEARCH

LITERATURE CITED

Arnett, E.B. and R. Sallabanks. 1998. Land man- Diamond, J. 1975. The island dilemma: lessons of ager perceptions of avian research and information modern biogeographic study for the design of nature needs in the U.S. Pacific Northwest. Pages 399-413 reserves. Biological Conservation 7:129-146. in J.M. Marzluff and R. Sallabanks (editors). Avian conservation: research and management. Island Dickson, J.G., F.R. Thompson, R.N. Connor, and Press, Covelo, California. K.E. Franzreb. 1993. Effects of silviculture on neotropical migratory birds in central and southeast- Babbie, A. 1995. The practice of social research. 7th ern oak pine forests. Pages 374-385 in Status and edition. Wadsworth, Belmont, California. management of neotropical migratory birds (D.M. Finch and P.W. Stangel, editors). USDA Forest Bailey, R.G. 1995. Description of the ecoregions of Service General Technical Report RM-229. Rocky the United States. USDA Forest Service Miscella- Mountain Forest and Range Experiment Station, neous Publication No. 1391. U.S. Department of U.S. Department of Agriculture, Fort Collins, Colo- Agriculture, Forest Service, Washington, DC. rado.

Barber, D.R., T.E. Martin, M.A. Melchiors, R.E. Thill, Donovan, T.M., C. Beardmore, D. Bonter, J. Brawn, and T.B. Wigley. 2001. Nesting success of birds in J. A. Fitzgerald, R. Ford, S. Geautreax, L. George, different silvicultural treatments in southeastern U.S. C. Hunter, T.E. Martin, J. Price, K. Rosenberg, P. pine forests. Conservation Biology 15:196-207. Vickery, and T.B. Wigley. 2002. Priority research needs for conservation of Neotropical migrants. Journal of Field Ornithology 73:329-338. Benson, J. and W.B. Michael. 1987. Designing evaluation studies: A twenty-year perspective. International Journal of Educational Research 11: Evans, K.E. and R.N. Conner. 1979. Snag manage- 43–55. ment. Pages 214-225 in Management of North Central and Northeastern forests for nongame birds Bonny, R., D.N. Pashley, R.J. Cooper, and L. Niles (R.M. DeGraaf and K.E. Evans, compilers). USDA (editors.). 2000. Strategies for bird conservation: the Forest Service General Technical Report NC-51. Partners in Flight planning process. Proceedings of North Central Forest Experiment Station, U.S. the 3rd Partners in Flight Workshop. Cape May, Department of Agriculture, St. Paul, Minnesota. New Jersey. USDA Forest Service Proceedings RMRS-P-16. Rocky Mountain Research Station, Finch, D.M. 1991. Population ecology, habitat U.S. Department of Agriculture, Ogden, Utah. requirements, and conservation of neotropical migratory birds. USDA Forest Service General Butcher, G.S. et al. 1992. Needs and assessment: Technical Report RM-205. Rocky Mountain Forest monitoring neotropical migratory birds. Partners in and Range Experiment Station, U.S. Department of Flight, Monitoring Working Group. Arlington, Vir- Agriculture, Fort Collins, Colorado. ginia. [Unpublished document] Finch, D.M. and P.W. Stangel. 1993. Status and Case, R., M. Andrews, and W. Werner. 1988. How management of neotropical migratory birds. USDA can we do it? An evaluation training package for Forest Service General Technical Report RM-229. development educators. InterAction American Rocky Mountain Forest and Range Experiment Council for Voluntary International Action, New York, Station, U.S. Department of Agriculture, Fort Collins, New York. Colorado.

56 LITERATURE CITED

Fink, A. 1995. How to ask survey questions. Sage Martin, T.E. and D.M. Finch. 1995. Ecology and Publications, Thousand Oaks, California. management of neotropical migratory birds: a synthesis and review of critical issues. Oxford Gram, W.K., P.A. Porneluzi, R.L. Clawson, J. University Press, New York, New York. Faaborg, and S.C. Richter. 2003. Effects of experi- mental forest management on density and nesting Martin, T.E., C.R. Paine, C.J. Conway, W.M. success of bird species in Missouri Ozark forests. Hochachka, P. Allen, and W. Jenkins. 1997. BBIRD Conservation Biology 17:1324-1337. Field Protocol. Montana Cooperative Wildlife Re- search Unit, University of Montana, Missoula. Hagan, J.M. and R. Boone. 1996. Harvest rate, configuration, and forest fragmentation: a simulation McMahon, J.P. 1992. Forest industry’s commitment of the 1989 Maine Forest Practices Act. Manomet to the public. Journal of Forestry 90:38-40. Technical Report Series MCDCF-97001. Manomet Center for Conservation Sciences, Manomet, McNamara, C. 1998. Basic guide to program Massachusetts. evaluation. [online at: http://www.mapnp.org/library/ evaluatn/fnl_eval.htm] Hagan, J.M. and D.W. Johnston (editors). 1992. Ecology and Conservation of neotropical migrant landbirds. Smithsonian Institution Press, Washing- Meffe, G.K. 1998. Conservation biology: into the ton, DC. millennium. Conservation Biology 12:1-3.

Hagan, J.M., P.S. McKinley, A.L. Meehan, and S.L. Morrissey, W.A., J.A. Zinn, and M.L. Corn. 1994. Grove. 1997. Diversity and abundance of landbirds Ecosystem management: federal agency activities. in a northeastern industrial forest. Journal of Wildlife Congressional Research Service Report 94-339 Management 61:718-735. ENR to Congress. Congressional Research Service, Washington, DC. Hagan, J.M., R.G. Wagner, and H. Daniel. 2002. Information priorities for forest structure: results of National Fish and Wildlife Foundation. 2002. 2001 the Forest Ecosystem Information Exchange survey. annual report. National Fish and Wildlife Founda- Mosaic Science Notes 2002-1. Manomet Center for tion, Washington, DC. Conservation Sciences, Manomet, Massachusetts. National Fish and Wildlife Foundation. 2003. 2002 Hannon, S.J. and McCallum. 2003. Using the focal annual report. National Fish and Wildlife Founda- species approach for conserving biodiversity in tion, Washington, DC. landscapes managed for forestry. Sustainable Forest Management Network Synthesis Paper. Edmonton, National Science Foundation. 1998. User-friendly Alberta, Canada handbook for project evaluation: science, mathemat- ics, engineering and technology education. Publica- Hilborn, R. 1987. Living with uncertainty in resource tion NSF93152. National Science Foundation, management. North American Journal of Fisheries Arlington, Virginia. Management 7:1-5. National Science Foundation. 2002. Long-Term Independent Sector. 2000. Public trust and account- Ecological Research Program Twenty-Year Review. ability. Independent Sector, Washington, DC. National Science Foundation, Arlington, Virginia.

International Union for the Conservation of Nature, Office of Management and Budget. 1995. Primer on United Nations Environment Programme, and World performance measurement. Office of Management Wide Fund for Nature. 1991. Caring for the earth: a and Budget, Washington, DC. [online at: http:// strategy for sustainable living. International Union for govinfo.library.unt.edu/npr/library/resource/ the Conservation of Nature, Gland, Switzerland. gpraprmr.html]

Kessler, W.B., H. Salwasser, C.W. Cartwright, Jr., Orosz, J.J., C.C. Phillips, and L.W. Knowlton. 2003. and J.A. Caplan. 1992. New perspectives for Agile philanthropy: understanding foundation sustainable natural resources management. Eco- effectiveness. Philanthropic and Nonprofit Knowl- logical Applications 2:221-225. edge Management Series, Monograph No. 1.

57 EVALUATION OF FOREST RESEARCH

Dorothy A. Johnson Center for Philanthropy and Terborgh, J. 1989. Where have all the birds gone? Nonprofit Leadership, Grand Valley State University, Princeton University Press, Princeton, New Jersey. Allendale, Michigan. Thompson, J.L., R.A. Lancia, and J.A. Gerwin. Richter, B.D. and K.H. Redford. 1999. The art (and 2002. Breeding biology of Swainson’s Warblers in a science) of brokering deals between conservation managed South Carolina bottomland. Third North and use. Conservation Biology 13:1235-1237. American Ornithological Congress. New Orleans, Louisiana. [Abstract] Rivera-Milan, F.F., R. Vides, G. Wong, and C. Hays. 1994. Management and monitoring of resident and Wigley, T.B. and J.M. Sweeney. 1993. Cooperative migratory birds and their habitats in Latin America partnerships and the role of private landowners. and the Caribbean: an Interamerican perspective. Pages 39-44 in Status and management of neotro- U.S. Fish and Wildlife Service, Office of International pical migratory birds (D.M. Finch and P.W. Stangel, Affairs, Arlington, Virginia. editors). USDA Forest Service General Technical Report RM-229. Rocky Mountain Forest and Range Robbins, C.S., J.R. Sauer, R.S. Greenberg, and S. Experiment Station, U.S. Department of Agriculture, Droege. 1989. Population declines in North Ameri- Fort Collins, Colorado. can birds that migrate to the Neotropics. Proceed- ings of the National Academy of Science 86:7658- Wilde, J. and S. Sockey. 1995. Evaluation hand- 7662. book. Evaluation Assistance Center, New Mexico Highlands University, Albuquerque, New Mexico. Romesberg, H.C. 1981. Wildlife science: gaining [online at: http://www.ncela.gwu.edu/miscpubs/ reliable knowledge. Jornal of Wildlife Management eacwest/evalhbk.htm] 45:293-313. Williams, J.E. and J.N. Rinne. 1992. Biodiversity Sherry, T.W. and R.T. Holmes. 1995. Summer management on multiple-use federal lands: an versus winter limitation of populations: what are the opportunity whose time has come. Fisheries 17:4-5. issues and what is the evidence? Pages 85-120 in Ecology and management of neotropical migratory birds: a synthesis and review of critical issues (T.E. Martin and D.M. Finch, editors). Oxford University Press, New York, New York.

Smith, W.B., J. Vissage, D. Darr, and R. Sheffield. 2001. Forest statistics of the United States, 1997. USDA Forest Service General Technical Report NC- 219. Northcentral Research Station, U.S. Depart- ment of Agriculture, St. Paul, Minnesota.

Stevens, J.P. 2001. Applied multivariate statistics for the social sciences. Lawrence Erlbaum Publishers, Mahwah, New Jersey.

Sustainable Forestry Board. 2002. Sustainable Forestry Initiative (SFI) Program, 2002-2004 edition. American Forest and Paper Association, Washing- ton, DC.

Suvedi, M. 2003. Introduction to program evalua- tion. Department of Agricultural and Extension Education, Michigan State University, East Lansing, Michigan. [online at: www.canr.msu.edu/evaluate/ sourcesfiles/introduction%20to%20PE.htm]

58 LITERATURE CITED

59