Report of the Committee of Visitors
Total Page:16
File Type:pdf, Size:1020Kb
Report of the Committee of Visitors Division of Physics - 2012 I- Introduction The Committee of Visitors (COV) for the Division of Physics of the National Science Foundation met during February 1-3, 2012, with the following charge: to review • The integrity and efficacy of processes used to solicit, review, recommend, and document proposal actions; • The quality and significance of the results of the Division’s programmatic investments; • The relationship between award decisions, program goals, and Foundation- wide programs and strategic goals; • The Division’s balance, priorities, and future directions; • The Division’s response to the prior COV report of 2009; and • Any other issues that the COV feels are relevant to the review. These charges were addressed in much detail, both by the committee at large as well as by the subpanels, comprised of a subset of the committee members and addressing each sub-discipline of the Division. The conclusions are presented in this report, which is divided into sections: I) introduction, II) the review process and the management of the Physics Division, III) some comments about the new programs, IV) societal benefits, V) special focus and IV) cyber infrastructure at the Physics Division, in addition to the required assessment. In addition, the committee has addressed all questions in the required report template that can be found as Appendix A to this report. Finally, the agenda for the three-day meeting, the list of participants, the membership of each panel and the detailed COV charges are presented as Appendices B-E. II. The review process and management of the Physics Division The committee was unanimous in their overall assessment that the management of the Physics Division was excellent. Much of this success was attributed to the leading team of Joe Dehmer and Denise Caldwell. Having two permanent, extremely qualified and committed people in the leadership of the Division makes all the difference. We understand that the NSF is moving towards having a substantial fraction of their officers to be rotators. While we understand the need for fresh blood to avoid stagnation and that the Division has benefited from excellent rotating program managers, NSF management should also recognize the challenge in 1 continually recruiting such high-quality individuals and the advantages provided by having greater continuity and more experienced program managers. 1. Quality and effectiveness of the review process There was consensus among the committee members that the three-tier review process is a critical component of excellence in review. We really like the fact that every program review process involved the request of outside reviews from experts on the respective proposals that was followed by a panel evaluation (which usually took place at the NSF). Finally, program directors made decisions based on these two factors. In addition, for larger proposals in programs such as Nuclear Physics, site visits are conducted by a panel of 3-5 independent reviewers for proposals by larger experimental groups (multi-PI awards of more than $1M). Although most panel conclusions were able to reconcile any differences that existed in the collection of individual reviews, we were particularly satisfied with how program directors were able to make decisions when there was no uniform consensus. They were able to consider all different points of view, take into account the PI funding history, and finally make a case for a final decision. A special review process was established for the Physics Frontier Centers where both external reviews and panel evaluations were much more detailed. Both merit review criteria are always addressed at all levels of evaluation (individual reviews, panel summaries, and program officer analysis). Most of the external reviewers demonstrated knowledge of the review process and the panels played an important role in balancing the importance of the two criteria. Some reviewers, however, had difficulty in properly weighting the “broader impacts” criterion. The level of detail in the postdoctoral mentoring plan varies significantly among proposals. Although much effort has been done in explaining these points to reviewers, we believe that an ongoing effort from the program directors educating the reviewers on both of these points is needed. There were two general recommendations. First, since the program officers’ evaluation makes a careful analysis of the review process and a clear discussion of the funding decision, we felt that this information, after the removal of any confidential data, should be made available to PI’s. This is especially important for PI’s that were close to the funding range and really could use this information in preparing their resubmission. Second, several sub-panels raised concern about award sizes. We feel that, in special cases of really great research, larger awards should be made. We understand that this will come at the expense of reducing the total number of awards. 2. Selection of reviewers There was most support for the quality of the reviewers selected. Some subpanels raised the concern that in some highly specialized areas it was very hard to find sufficient reviewers. Even when difficult, the committee felt that a minimum of 3 ad 2 hoc reviews should be requested for each proposal, and at least 2 ad hoc reviews should be received in each case. In the majority of the cases, the program satisfied these conditions. Some minor concerns about conflict of interest were raised. Although this was not a problem in the majority of the cases, some sub-panels observed several cases where conflicts of interest were not uncovered until the review process was under way. The program directors were able to respond quickly and address these issues in real time. 3. Program Balance Overall the subpanels were very satisfied with the program’s portfolio goals and balance. We were satisfied with the balance of renewals versus new investigators. Details can be found in the subpanel reports. 4. The COV process and response to previous COVs Most of the subpanels have been very satisfied with the responsiveness of the Physics Division handling the requests and recommendations of the previous COV report. Specific details can be found in the subpanel reports. The only major concern raised during the current COV was regarding the subpanel for computational physics. This panel also addressed the topic of cyber infrastructure, which also reviews computational physics. It was composed of members of other subpanels and so its activities caused some disruption to the activities of those subpanels, which were affected by not having their full membership during some deliberations. Other than that, everybody was extremely happy with the format of the COV and with what we were able to accomplish during these three days. III. Program portfolio and balance – creation of new programs. The Physics Division is composed of several science programs covering diverse areas of physics. According to the Division’s master plan, at least 50% of its budget should be allocated to grants to investigators funded by these programs. The actual percentage is slightly larger. Typical awards vary for different programs. Some programs tend to fund more groups of investigators, while other focus on single or few investigator awards. Even with these differences, the subpanels were very satisfied with the overall planning, development and management of the portfolio. The Physics Frontier Centers are allocated no more than 10% of the budget and currently they are using about 8%. The remainder of the budget is used to cover facilities and instrumentation resources, together with some small additional expenses. More recently, additional funds have been saved for the instrumentation program. For more details about this program, please read the report by the subpanel on APPI and the discussion below on Facilities and Instrumentation. 3 The committee paid special attention to the new programs and were clearly satisfied with most of what they have been achieving. The programs in particle and nuclear astrophysics have now been consolidated as the program in Particle Astrophysics and are clearly fulfilling a necessary scientific niche that was missing in the Physics Division portfolio. As can be seen from their subpanel report, this is a well-established program with great scientific achievements. The program in Physics of Living Systems has grown and has provided leadership for this new and interdisciplinary area of physics. We were particularly happy with the evolution of this program, an area of research that could overlap with programs in Biology and DMR Divisions. The program has a set of priorities for POLS that clearly distinguishes this program from work in other Divisions. First, the research must address the physics of living systems, not the use of biomolecules as materials (a purview of DMR). Second, to distinguish this program from BIO, the research must emphasize working on biological systems with a physics approach and must include a theoretical component that allows for quantification and for the development of new ideas. Biological research is, at times, mostly descriptive and cannot be easily generalized to other systems. There were concerns about the goals and breadth of the computational physics program and this concern is clearly raised in the subpanel report. Similarly to the POLS program, we would like to reinforce the need to extend the boundaries between fields that traditionally have not had strong interactions. We were also struck by the high level of relevance of AMO cold atom simulation experiments to significant problems in condensed matter physics, and the close collaborations between some CMP theorists and AMO experimental groups. We agree that this aspect should be encouraged. The AMO subpanel report provides more details on how to improve this connection. There was strong support for the Physics Frontier Centers program. This program clearly fosters major advances at the intellectual frontiers of physics and has been one of the flagships of the Division.