THE SCIENCE NEWS METRICS CAROL A. CHRISTIAN Space Telescope Science Institute Homewood Campus 3700 San Martin Drive Baltimore MD 21218, USA [email protected] AND GREG DAVIDSON Northrup Grumman Space and Technology One Space Park Drive Redondo Beach CA 90278, USA [email protected] Abstract. Scientists, observatories, academic institutions and funding agen- cies persistently review the usefulness and productivity of investment in scientific research. The Science News Metrics was created over 10 years ago to review NASA’s performance in this arena. The metric has been useful for many years as one facet in measuring the scientific discovery productivity of NASA-funded missions. The metric is computed independently of the agency and has been compiled in a consistent manner. Examination of the metric yields year-by-year insight into NASA science successes in a world wide context. The metric has shown that NASA’s contribution to world- wide top science news stories has been approximately 5% overall with the Hubble Space Telescope dominating the performance. 1. Introduction The Office of Space Science (OSS) of the US National Aeronautics and Space Administration (NASA), recently absorbed into NASA’s Science Mis- sion Directorate, is responsible for developing a space science program with a primary objective to accomplish fundamental science. Ultimately, like many science organizations, NASA’s success is measured in part by the achievement of scientific insights relative to the cost. NASA OSS consid- 2 CAROL A. CHRISTIAN AND GREG DAVIDSON ers scientific potential and output of missions in strategic planning and is held accountable for the associated costs. Independent measures of scientific accomplishments have been an integral part of this strategy. While fundamental science is the primary objective of the Space Science program, it is also among the most difficult of outcomes to measure. In the early 1990’s, one of us (Davidson) led an effort at NASA to identify science metrics. At the most fundamental level, a good metric is something that you can count which is correlated with what you want. Desirable secondary characteristics include ease of data collection, precision, and low levels of bias. The group at NASA explored both prospective measures (planned capabilities in terms of angular resolution, spectral resolution, sensitivity, and time resolution, all as a function of wavelength coverage) as well as retrospective measures (quantity of data, number of observations, and bib- liometric measures such as number of refereed papers or frequently cited refereed papers). Two retrospective metrics were selected from this analysis as being par- ticularly well-suited to address NASA strategic planning needs. Key advan- tages of these metrics included relative ease of collection, independence of NASA, correlation with other, more complex metrics (particularly the cita- tion bibliometrics) and ability to communicate results in a meaningful way to policy makers and to the public. These metrics are not perfect surrogate measures of all aspects of scientific performance, but they do provide im- portant insights into fundamental scientific performance. The first measure, colloquially referred to as the “Science News Metric” is based on the an- nual listing of “most important stories” in the journal, Science News. This listing has a 31-year history and is published at the end of each calendar year. The Science News Metric essentially tracks “what’s hot” in science on a year-by-year basis. The second metric formulated for OSS is the “Textbook Metric”. This measure is an attempt to penetrate how the “hot” science topics of a single year get incorporated in the body of knowledge. The purpose is to un- derstand OSS’s capture of “intellectual market share” (what percentage of textbook material is based on OSS contributions) in the long term as well as overall growth of knowledge about astronomy. This metric will not be discussed herein. A similar metric was discussed by Christian (2004) in evaluating the impact of the Hubble Space Telescope mission. 2. A Description of the Science News Metric Science News is published weekly. It summarizes scientific findings in fields as diverse archeology, biomedicine, chemistry, mathematics, psychology, space science, and technology. Science News captures the essence of ref- SCIENCE NEWS METRICS 3 ereed scientific publications in a digest form. Since 1973, Science News has published an annual list highlighting over 100 (usually 125-170) of the “most important stories” in science. Each year, the Science News Metric is compiled (by Davidson) to track those stories and estimate the OSS contribution to each. The metric represents the scientific or technical ac- complishments for each year and from this the performance of Space Science funded by NASA can be compared over to all other “world-class” science in fields as diverse as archeology to biomedicine. 2.1. CALCULATION The Science News Metric is calculated as follows: 1. All the “most important stories” are screened. Those that are not based on discoveries (data collected and scientific inferences made) or technological accomplishments are eliminated, usually about 15% of the total. 2. One point is awarded for each “most important story”. In most cases, the discovery is due to collaborativeefforts, and so credit is apportioned among the groups (foreign, ground-based astronomers, etc.) referenced by Science News as being responsible for the discovery. 3. NASA “points” are compared against points for all other scientific discoveries to establish NASA Space Science as a percentage of world science. The method allows comparison of NASA with other federal agencies (e.g., National Science Foundation). For example, in 2002, one of the top stories involved Dark Matter. In fact, Science News had three separate articles on this particular subject. The contributing NASA missions were the Wilkinson Microwave Anisotropy Probe (WMAP), the Hubble Space Telescope (HST), the High Energy As- tronomical Observatory (HEAO), and the Sloan Digital Sky Survey (SDSS), but with different contributions (WMAP being the most significant followed by HST). The discovery was definitely based on data and research results, and involved more that one agency. WMAP was given 0.38 points, HST given 0.25 points, HEAO given 0.03 points and the SDSS given 0.05 points. The total number of points allocated is less than 1.0 because non-space facilities contributed to this story. 2.2. STRENGTHS AND WEAKNESSES The strengths of the Science News Metric for the purpose intended are, first, that the data used to generate this metric have been determined inde- pendently of NASA. The news stories are selected by a completely indepen- dent entity. The research and assessment following the publication of the 4 CAROL A. CHRISTIAN AND GREG DAVIDSON year-end issue of Science News in December is conducted autonomously. Additionally, this metric provides data on individual missions as a function of time, thus yielding insight to support life-cycle cost trade-offs. One correlation with bibliometrics that was surprising to some in the early 1990’s was that 30%-40% of science return occurred after the comple- tion of prime mission lifetime. A parallel study of bibliometrics on IUE and OAO showed that peak publication of papers was 4-6 years after launch, and that peak publication of frequently cited papers (>5 citations/year) occurred 5-7 years after launch. Note there is some randomness in the metric due to the incidence of discoveries in a given calendar period. As an example, two black hole dis- coveries may share one point in one year. If the results had been split between years, the stories might have been attributed to more than one point. As an additional caveat, it is recognized that the Science News Metric is based on the journal Science News, and not as rigorous as a refereed sci- entific journal. One can argue that the number of “most important stories” from a mission as highlighted by a commercial journal is not necessarily correlated with the scientific value of that mission. What researchers value as the most important advances in their discipline may loosely correlate with what Science News reports, but the correspondence is not necessarily one-to-one. Also different science stories may involve varied levels of effort. For example, the importance of the single finding of the origin of the uni- verse based on the Cosmic Background Explorer (COBE) mission data and analysis took considerable time and effort. In some sense the Science News Metric can capture the importance and level of effort in such situations because, for example, the COBE result was so significant that the story re-emerged repeatedly in subsequent years as a “most important story”. Alternatively, attempts at using refereed publications on a year-to-year basis to evaluate the impact of specific discoveries and research results is time consuming and problematic. Usually refereed publications take consid- erable time to reach publication and an important subject may take years to accumulate the representative articles that would indicate “importance”. Using a “subject citation index” if it existed as such would also be difficult in that it takes several years for a specific set of results to be assimilated by the scientific community and then referenced. It follows that refereed publications and citations can be useful for retrospectives, but as timely measures of productivity they are risky. The importance of the Science News Metric should be understood. As one facet of the accomplishments that NASA uses to formally report ac- tivity under the Government Performance and Results Act (GPRA), this indicator clearly is taken as a serious measure. The metric can have some SCIENCE NEWS METRICS 5 deliberate influence on the funding for missions and specific lines of sci- entific inquiry. Certainly missions that appear favorably with high Science News Metrics use the metrics as a leverage point in arguing for continuation or augmentation.