Diction Model Output – a Perspective Based on Verifi- Cation of Temperature and Precipitation Guidance

Home , Model output statistics

5.8 OBJECTIVE INTERPRETATION OF NUMERICAL WEATHER PRE- DICTION MODEL OUTPUT – A PERSPECTIVE BASED ON VERIFI- CATION OF TEMPERATURE AND PRECIPITATION GUIDANCE

J. Paul Dallavalle* and Valery J. Dagostaro Meteorological Development Laboratory Office of Science and Technology National Weather Service/NOAA Silver Spring, Maryland

1. INTRODUCTION of public and aviation weather services provided by the NWS. Forecasts produced by In the late 1960’s and early 1970’s, the each NWS forecast office are evaluated for National Weather Service (NWS) began im- one or more cities in the office’s area of re- plementing guidance products that objectively sponsibility, and the skill of the human fore- interpreted the output of numerical weather cast is compared to that provided by the guid- prediction (NWP) models. Initially, these ance for the same weather element. Thus, products were for weather elements such as verifications of the max/min temperature and maximum (max) and minimum (min) tem- PoP provide one look at the evolution of skill perature or the probability of precipitation in the NWP models and the NWS public (PoP) during 12-h periods. The first interpre- weather forecasts from 1966 to the present. tive products were based on the “perfect prog” (Klein and Lewis 1970) statistical technique. Earlier studies have examined trends in the By 1973, the Model Output Statistics (MOS) skill of the NWS forecasts. Zurndorfer et al. approach (Glahn and Lowry 1972) had super- (1979) looked at verifications of guidance and ceded the perfect prog method, and the num- local public weather forecasts for the period of ber of interpretive guidance products in- 1970 through 1977 and found improvements creased rapidly. Since then, the MOS ap- in both the PoP and max/min temperature proach has been used by the NWS to generate forecasts. Charba and Klein (1980) found a guidance products from the Primitive Equa- substantial increase in skill over a 10-year pe- tion, Limited-area Fine-mesh, Nested Grid, riod when verifying the PoPs of the local Eta, and Global Forecast System models. NWS offices for projections of 24-36 and 36- 48 hours in advance. Ramage (1982) contra- Much of the improvement in NWP and in dicted these results and claimed that, except in the statistical interpretation system can be certain regions of the U.S. in winter, any in- tracked by the verification of the weather ele- crease in accuracy of the PoPs from 1966 to ment guidance. Since 1966, the NWS has 1978 was negligible. Glahn (1985), however, maintained a national verification program in found significant improvement in the local which selected NWS forecast products are NWS PoPs over the 15-year period from 1967 routinely verified as an indicator of the quality to 1982. Carter and Polger (1986) showed that the national skill scores for both the local *Corresponding author address: J. Paul NWS and the guidance PoP had significantly Dallavalle, 1325 East-West Highway, Sta- increased since 1966 for all seasons, both tion 11306, Silver Spring, MD 20910-3283; forecast cycles, and all projections examined. e-mail:[email protected] Murphy and Sabin (1986) confirmed the Carter and Polger study and found that on the Because the NWP models were, at first, national level, the NWS PoP and max/min run only from 0000 and 1200 UTC initial temperature forecasts were of significantly conditions, and because the forecasts issued to higher quality by the end of the approximately the public were based on these runs, only 15-20 year period studied than in 1966. forecasts from two issuance times were verified. The public forecasts were disseminated In this paper, we examine the skill of the at approximately 0400 and 2000 local time official NWS max/min and PoP forecasts (LT) and contained information for three fore- compared to the guidance for forecast periods cast periods. From the 0400 LT issuance out to approximately 60 hours in advance. (0000 UTC model cycle), the public weather After a brief description of the evolution of forecasts were valid for today, tonight, and the NWS verification program for temperature tomorrow. For PoP, these definitions corre- and PoP, we describe changes made to the sponded to forecasts valid 12-24, 24-36, and guidance system during the period of record, 36-48 hours after 0000 UTC. For the summarize the methodology used to verify the max/min temperature, the forecasts were valid guidance and local forecasts, and show a for today’s max, tonight’s min, and tomor- number of verification time series beginning row’s max, which nominally were valid ap- in the late 1960’s. As expected, the skill of proximately 24, 36, and 48 hours after the NWS max/min temperature and PoP fore- 0000 UTC. Similarly, from the 2000 LT issu- casts, as well as that of the guidance, has in- ance (1200 UTC cycle), the public weather creased significantly since the late 1960’s and forecasts were available for tonight, tomor- early 1970’s. While the increase in skill of the row, and tomorrow night. For the PoPs, these day 1 forecast has leveled off during the last projections were defined exactly as for the decade, the skill of the day 2 forecast provided 0000 UTC cycle, namely, valid for 12-h peri- to the public is now comparable to that of the ods ending 24, 36, and 48 hours after initial day 1 forecast provided 20 years ago. model time. For the max/min temperature, the forecasts were for tonight’s min, tomorrow’s 2. PUBLIC WEATHER VERIFICATION max, and tomorrow night’s min, and nominally were valid approximately 24, 36, and The National Weather Service program to 48 hours after 1200 UTC. A fourth period for verify forecasts issued to the public is de- the max/min temperatures was added in Octo- scribed in Carter and Polger (1986) and Da- ber 1975 so that an additional min (max) tem- gostaro et al. (1989). Established in 1966, the perature became available for the 0000 (1200) verification program was designed to evaluate UTC cycle. The fourth period PoP was not the quality of the max/min temperature and available to the verification system until PoP forecasts issued by the local NWS fore- summer 2002. cast offices and to compare those forecasts with the guidance provided to the offices. In During the verification period, four differ- 1966, the guidance was provided by forecast- ent approaches to data collection were used. ers at the National Meteorological Center From 1966 until March 1974, extensive hu- (NMC), now the National Centers for Envi- man labor, either at the forecast office or at ronmental Prediction (NCEP). Within several NWS Headquarters, was required to record the years, objective guidance obtained by apply- local forecasts and verifying observations on a ing statistical methods to NWP model output form and transfer these values to punched replaced the subjective guidance. cards. The subjective guidance was also obtained from a form and punched cards; when the guidance became objective, the appropri- transmitted back to the NWS central computer ate guidance was extracted from the NWS system. Subsequently, staff from the Tech- central computer system. In March 1974, a niques Development Laboratory (now the Me- second phase was initiated in which more ex- teorological Development Laboratory or tensive use was made of machine-based data MDL) processed the data, did quality-control, extraction methods. For 15 months, the local and provided periodic verifications and analy- forecasts and verifying observations were en- ses. Ruth and Alex (1987) described the AEV tered on mark sense cards which could be read program; Dagostaro (1985) described the sys- directly by a computer. By July 1975, data tem used to process the AEV data. collection became totally automated when the NWS implemented software to collect the lo- In January 2000, the AEV program was cal max/min and PoP forecasts from the transferred to the NWS Advanced Weather Coded City Forecast bulletin and the verifying Interactive Processing System (AWIPS). The observations from the Selected Cities Sum- AWIPS verification program (AVP) was mary prepared by the National Climatic Data analogous to the AEV software. Further Center. The guidance continued to be taken changes in the NWS methodology of generat- from files on the NWS central computer sys- ing forecast products, specifically, the intro- tem. duction of the Interactive Forecast Preparation System (IFPS) and the generation of local A third era of data collection began after forecasts on grids, began to eliminate the local the NWS introduced the Automation of Field use of the AVP software. This fourth era of Operations and Services (AFOS) system into data collection was starting to draw to a close its field operations. In October 1983, the by the spring of 2004. NWS implemented the AFOS-era verification (AEV) program. This program, run locally at 3. GUIDANCE AVAILABILITY each NWS forecast office, created a verification database in which the appropriate local When the verification program began in forecasts for the max/min temperature and 1966, NMC forecasters provided subjective PoP, the verifying observations, and the objec- PoP and max/min temperature guidance to the tive guidance were stored. As before, the lo- local forecast offices. Until April 1969, for cal forecasts were extracted from the Coded the 12-24 h PoP, the guidance was categorical, City Forecast bulletin. The verifying observa- that is, the PoP was equal to 0 or 100%, for no tions were obtained from a local database of rain or rain, respectively. The PoPs for the hourly and synoptic observations. The guid- other forecast projections were probabilities ance was obtained from the appropriate MOS that ranged from 0 to 100%. From April 1969 alphanumeric bulletin. The AEV program to the present, the PoP guidance for all projec- was run twice daily to pick up the appropriate tions presented a range of probabilities. In forecast information and to collate forecasts January 1972, the subjective PoP guidance with the verifying observations. The local produced by human forecasters was replaced forecaster had an editing capability so as to by a MOS PoP system (Lowry and Glahn modify any local forecasts or observations 1976) developed from the Primitive Equation which were stored erroneously in the database. (PE) and trajectory models. Since that time, Within 5 to 7 days of the generation of the the official PoP guidance issued to the local original forecast, an alphanumeric bulletin forecasters has been based on the MOS ap- containing the collated local forecasts, objec- proach applied to one of the NWP models. tive guidance, and verifying observations was Similarly, for the max/min temperature, the GFS-based MOS guidance became the of- subjective guidance was issued by NMC fore- ficial guidance for the purpose of verification. casters until 1970. In April 1970, the NWS implemented objective max/min temperature One last change in the characteristics of guidance based on the perfect prog approach the guidance must be noted. Although fore- applied to the Primitive Equation (PE) model. casters were generating public weather tem- In August 1973, the perfect prog guidance was perature forecasts that referred to daytime and replaced by a MOS max/min temperature nighttime periods, the original MOS guidance guidance system developed from the PE and for max/min temperature was valid for a cal- trajectory models. Like the PoP guidance, the endar day max and min. This valid period official max/min temperature guidance since was dictated by the availability of the observa- that time has been generated by the MOS ap- tions used in the MDL developmental system. proach applied to one of the NWP models. In November 1985, MDL implemented a new MOS system that predicted daytime max and As noted, the NWS was using the MOS nighttime min temperatures (Erickson and approach extensively by 1973. In subsequent Dallavalle 1986). As will be seen in the veri- years, different packages of MOS guidance fication time series, this particular change re- were often available, depending on the avail- sulted in substantial improvement in the accu- ability of NWP models. For many of the last racy of the guidance. 30 years, the forecasters have had access to two or more guidance packages when produc- 4. VERIFICATION METHODOLOGY ing the public forecast. For instance, in 2004, forecasters can use for the first four forecast In doing this latest verification study, periods max/min temperature and PoP guid- we’ve verified the local forecasts and the ance that is based on the Nested Grid Model guidance since April 1966. A matched sample (NGM), the Global Forecast System (GFS) of local forecasts and guidance was used. model, or the Eta model. Obviously, this Verifications were done for two seasonal plethora of guidance complicates comparative stratifications, namely, for warm (April 1 – verification. For the NWS verifications pre- September 30), and cool (October 1 – sented in this paper, however, only the “offi- March 31) seasons. We evaluated the local cial” guidance is used. The choice of the offi- forecasts and guidance for both cycles (0000 cial MOS guidance product was dictated by and 1200 UTC) and all projections. In the re- the software collecting the MOS guidance, sults shown here, the verification scores for and not by the newest or most accurate guid- the PoPs for a specific projection, for exam- ance package available. As noted above, the ple, 12-24 hours after 0000 or 1200 UTC, are official MOS guidance was initially PE-based. combined to produce one set of scores. In April 1980, the MOS guidance based on the Though verification results are highly variable Limited-area Fine-mesh model (LFM) became between the two seasons, we did not find that the official guidance package. The NGM- the PoP scores varied as much between the based guidance replaced the LFM-based guid- daytime and nighttime period and so com- ance in June 1993 as the official package. Fi- bined them. In contrast, the verifications of nally, during the summer of 2002 (the precise the max and min temperatures vary substan- date varied from station to station because of tially and so we did not combine the scores for the difficulty of implementing software in a those two elements. distributed processing system like AWIPS), The PoP forecast represents the probability (Fig.1) were verified. Results are shown here of measurable precipitation (defined as for all stations combined. 0.01 inches or more) occurring within a specific 12-h period ending at either 0000 or A number of verification measures were 1200 UTC. Thus, the verifying observation is computed. For the max/min temperature, we available by summing two 6-h precipitation calculated mean absolute error, bias or mean amounts. In contrast, the max (min) tempera- algebraic error, root mean square error, and ture forecast represents the high (low) tem- the percent improvement of the forecasts com- perature occurring during the local day pared to a forecast based on the normal (night). Observations of the max/min tem- max/min temperature. We used the 1961-90 perature taken at standard observing sites co- normals obtained from the National Climatic incide with a UTC interval or with a local cal- Data Center as our climatic standard. Only endar day. Thus, no direct observation is the mean absolute error statistics are discussed made of the daytime max or nighttime min. here. Prior to the 1984-85 cool season, the daytime max was estimated by using the maximum For PoP, the Brier score (Brier 1950) and temperature reported for the 1200-0000 UTC the relative improvement over the Brier score period. The nighttime min was estimated by obtained by using the appropriate monthly the 0000-1200 UTC report of the minimum climatic relative frequency as the forecast for stations in the eastern and central U.S., and were calculated. Note that the Brier score is by the 0000-1800 UTC report of the minimum equal to the mean square error for a probabil- for stations in the western part of the U.S. istic forecast and a verifying observation of Starting with the 1984-85 cool season, an al- either 0 or 1 for no precipitation or precipita- gorithm was implemented in the AEV system tion, respectively. The climatic relative fre- to estimate the daytime max (defined as quencies used for the percent improvement 7 a.m. - 7 p.m. Local Standard Time) and the score were based on precipitation records for nighttime min (defined as 7 p.m. - 8 a.m. Lo- the period of 1972 through 1985 (Jensenius cal Standard Time). Enhancements to this al- and Erickson 1987). In addition, we calcu- gorithm were made in subsequent years as ob- lated the reliability for the PoP which meas- serving standards were modified. ured the correspondence between a particular probability forecast and the relative frequency During the period from 1966 to 2004, the of the event. number of stations available for verification varied as a function of the NWS organiza- tional structure and the process being used to 5. RESULTS collect the data. To establish a relatively con- stant set of sites, we checked station availabil- Figures 2 and 3 show the mean absolute ity during three periods of time, namely, the error of the local and guidance forecasts for pre-AEV era (1966-1983), the AEV era today’s (24 hr) and tomorrow’s (48 hr) max (1983-1999), and the AVP era (2000-2004). temperature for the warm and cool seasons, To be included in the long-term verifications, respectively. As discussed previously, these a station had to be available during all three forecasts were generated during the 0000 UTC periods, and we had to have approximately cycle. Several features can be noted. First, 50% or more of all possible data from each of the cool season errors are greater than the the three periods. Under these constraints, warm season errors for the same projection. forecasts for 80 stations in the contiguous U.S. Second, the cool season errors seem to exhibit

Figure 1. Sites for which local and guidance forecasts were verified. Shading denotes the NWS administrative regions.

7 Guid max temp 24 hr Local max temp 24 hr 6.5 Guid max temp 48 hr Local max temp 48 hr 6

5.5

4.5 MAE (degMAE F) 4

3.5

2.5

2 1966 1969 1972 1975 1978 1981 1984 1987 1990 1993 1996 1999 2002 Year

Figure 2. Mean absolute error for warm season local and guidance forecasts of today’s (24-hr) and tomorrow’s (48-hr) max temperature generated during the 0000 UTC cycle.

7 Guid max temp 24 hr Local max temp 24 hr 6.5 Guid max temp 48 hr Local max temp 48 hr 6

5.5

4.5 MAE (degF) 4

3.5

2.5

2 1966 1969 1972 1975 1978 1981 1984 1987 1990 1993 1996 1999 2002 Year

Figure 3. Same as Fig. 2, except for cool season verifications.

more variability from one season to another Figures 4 and 5 show for the warm and than the warm season errors. Third, local cool seasons, respectively, the verification forecasters are consistently more accurate than time series for tomorrow’s (36-hr) and the day the guidance, and this improvement is rela- after tomorrow’s (60-hr) max temperature tively consistent in the warm season. In the forecasts generated during the 1200 UTC cy- cool season, the variability in the improve- cle. The results are similar to those of Figs. 2 ment of the local forecast relative to the guid- and 3. Note the striking improvement in the ance is somewhat greater. Fourth, the general accuracy of the 60-h max temperature guid- improvement in both the local forecasts and ance for the 2002-03 and 2003-04 seasons co- the guidance over the period of record is evi- incident with the use of the GFS guidance as dent. For instance, the accuracy of the local the “official” product. forecast for tomorrow’s max is now as accurate as the forecast for today’s max was Mean absolute errors for tonight’s (24 hr) 20 years ago. The guidance during the cool and tomorrow’s (48 hr) min temperature fore- season for tomorrow’s max is now as accurate casts generated during the 1200 UTC cycle as the guidance for today’s max was 10 years are shown in Figs. 6 and 7, respectively, for ago. The downward trend in error is greater in the warm and cool seasons. As with the veri- the cool season. Finally, the improvement in fications for the max temperature, several fea- the guidance due to the implementation of the tures are notable. First, the improvement in MOS approach (August 1973) is quite obvi- the quality of the local forecasts and the guid- ous, particularly during the cool season. ance over the years is evident. For instance,

7 Guid max temp 36 hr Local max temp 36 hr 6.5 Guid max temp 60 hr Local max temp 60 hr 6

5.5

4.5 MAE (deg F) MAE (deg 4

3.5

2.5

2 1966 1969 1972 1975 1978 1981 1984 1987 1990 1993 1996 1999 2002 Year

Figure 4. Same as Fig. 2, except for tomorrow’s max (36-hr) and the day after tomorrow’s max (60-hr) generated during the 1200 UTC cycle.

7 Guid max temp 36 hr Local max temp 36 hr 6.5 Guid max temp 60 hr Local max temp 60 hr

5.5

4.5 MAE (degF) 4

3.5

2.5

2 1966 1969 1972 1975 1978 1981 1984 1987 1990 1993 1996 1999 2002 Year

Figure 5. Same as Fig. 4, except for cool season.

7 Guid min temp 24 hr Local min temp 24 hr 6.5 Guid min temp 48 hr Local min temp 48 hr 6

5.5

4.5 MAE (degF) 4

3.5

2.5

2 1966 1969 1972 1975 1978 1981 1984 1987 1990 1993 1996 1999 2002 Year

Figure 6. Mean absolute error for warm season local and guidance forecasts of tonight’s (24-hr) and tomorrow night’s (48-hr) min temperature generated during the 1200 UTC cycle.

7 Guid min temp 24 hr Local min temp 24 hr 6.5 Guid min temp 48 hr Local min temp 48 hr 6

5.5

4.5 MAE (deg F) 4

3.5

2.5

2 1966 1969 1972 1975 1978 1981 1984 1987 1990 1993 1996 1999 2002 Year

Figure 7. Same as Fig. 6, except for the cool season.

the accuracy of the local forecast of tomorrow creased substantially. After the 1984 warm night’s min is now as good as the accuracy of season, the accuracy of the local forecasts and tonight’s min was 20 years ago. Similarly, the the guidance seems relatively invariant from errors of the guidance for tomorrow night‘s year to year, except for a slow downward min are now about the same as the errors of trend in the errors. In contrast, during the cool the guidance for tonight’s min about 10 years season, the inter-annual variability in the er- ago. Second, unlike the max temperature rors is noticeably less after the 1985-86 cool forecasts, the difference in accuracy between season, and the improvement in the errors of the local forecasts and the objective guidance the 48-h local forecasts and guidance is pro- is very small during the warm season, and has nounced. become increasingly smaller during the cool season. Third, the implementation of the Figures 8 and 9 show the verification time MOS approach in August 1973 seemed to im- series for tomorrow’s min (36-h) and tomor- prove the warm season guidance. A similar row night’s min (60-h) generated during the improvement was not evident in the cool sea- 0000 UTC cycle for the warm and cool season guidance. Fourth, the implementation of sons, respectively. As with the 60-h max the nighttime min guidance in late 1984 temperature guidance, the improvement in the seemed to have the biggest impact on the er- guidance for tomorrow night’s min is quite rors of both the local forecasts and the guid- dramatic after changes were made during the ance. The average error was lower after that 2002 summer to use the GFS-based MOS time, and the inter-annual variability de- guidance as the official guidance package.

7 Guid min temp 36 hr Local min temp 36 hr 6.5 Guid min temp 60 hr Local min temp 60 hr 6

5.5

4.5 MAE (deg F) (deg MAE 4

3.5

2.5

2 1966 1969 1972 1975 1978 1981 1984 1987 1990 1993 1996 1999 2002 Year

Figure 8. Same as Fig. 6, except for tonight’s min (36-hr) and tomorrow night’s min (60-hr) generated during the 0000 UTC cycle.

6.5

5.5

4.5 MAE (deg F) 4

3.5

3 Guid min temp 36 hr Local min temp 36 hr 2.5 Guid min temp 60 hr Local min temp 60 hr 2 1966 1969 1972 1975 1978 1981 1984 1987 1990 1993 1996 1999 2002 Year

Figure 9. Same as Fig. 8, except for the cool season.

The Brier skill score (defined as the im- and the guidance is quite dramatic, particu- provement in the Brier score of the forecasts larly in the cool season. As was seen in the or guidance, relative to a Brier score based temperature verifications, the skill of the on a prediction of the climatic relative fre- local forecasts for the 48-h PoP is now about quency) is shown in Figs. 10 and 11 for the as great as the skill of the 24-h PoP was warm and cool seasons, respectively. Only 20 years ago. Similarly, the skill of the 48-h the scores for the 12-24 and 36-48 h projec- PoP guidance is now about the same as the tions for the local and guidance forecasts are skill of the 24-h PoP guidance 10 years ago. shown. Forecasts from both the 0000 and Third, the correlation in skill between the 1200 UTC cycles were combined in the local forecasts and the guidance is very verifications. Note that the Brier skill score strong. Fourth, while the local forecasts are for the 24-h guidance prior to 1969 is mean- generally more skillful than the guidance, in ingless because only categorical guidance, some seasons (for example, the 2002-03 and rather than probabilities, were available. 2003-04 cool seasons and the 2003 warm Several features of the verifications are no- season), the 48-h guidance is more skillful table. First, the skill scores are much higher than the local forecasts. Lastly, the correla- in the cool season than in the warm season. tion between the improvement of the local Thus, by the 2003-04 cool season, the Brier forecasts or the guidance and a specific skill scores for the 48-h PoP are actually model implementation or change in the sta- higher than the warm season skill scores for tistical approach does not seem high. the 24-h PoP. Second, the improvement in Rather, the improvements tend to reflect the the skill scores for both the local forecasts overall improvement in the NWP system.

0.7

Guid POPS 24 hr Local POPS 24 hr

0.6 Guid POPS 48 hr Local POPS 48 hr

0.5

0.4

0.3

Brier ScoreBrier Improvement over Climate 0.2

0.1

0 1966 1969 1972 1975 1978 1981 1984 1987 1990 1993 1996 1999 2002 Year

Figure 10. Improvement in Brier score of local and guidance PoP’s for the 12-24 h (24hr) and 36-48 h (48 hr) forecasts during the warm season. Results from both cycles were combined.

0.7

Guid POPS 24 hr Local POPS 24 hr

0.6 Guid POPS 48 hr Local POPS 48 hr

0.5

0.4

0.3

0.2 BrierScore Improvement over Climate

0.1

0 1966 1969 1972 1975 1978 1981 1984 1987 1990 1993 1996 1999 2002 Year

Figure 11. Same as Fig. 10, except for the cool season. . 6. CONCLUSIONS and Kari Sheets who provided the map indi- cating the verification sites. As the results indicate, the improvement in the quality of the local forecasts and the guid- 8. REFERENCES ance is quite obvious in the verification time series. In general, the local forecasts for day 2 Brier, G. W., 1950: Verification of forecasts are as good as they were for day 1 about expressed in terms of probabilities. Mon. 20 years ago. The quality of the guidance for Wea. Rev., 78, 1-3. day 2 is about as good as the guidance for day 1 was 10 years ago. Improvements for the Carter, G. M., and P. D. Polger, 1986: A 20- 48- and 60-h forecasts are particularly obvi- year summary of National Weather Ser- ous. In general, temperature forecasts during vice verification results for temperature the cool season have larger errors than during and precipitation. NOAA Tech. Memo. the warm season, but also show the greater NWS FCST-31, 50 pp. improvement in skill over the period. The PoP forecasts have the greatest skill during the Charba, J. P., and W. H. Klein, 1980: Skill in cool season when convection and mesoscale precipitation forecasting in the National systems are less common. Weather Service. Bull. Amer. Meteor. Soc., 61, 1546-1555. To understand the improvement in the overall forecast system since the late 1960’s, Dagostaro, V. J., 1985: The national AFOS- we will eventually add other verification era verification data processing system. measures to this study. For instance, the sea- TDL Office Note 85-9, 47 pp. [Available sonal verification scores can be fit with re- from NWS Meteorological Development gression lines to determine trends in im- Laboratory, 1325 East-West Highway, provement. The Brier score can be decom- Silver Spring, MD 20910-3283.] posed into its components to assess whether forecast reliability or resolution has increased. Dagostaro, V. J., C. L. Alex, G. M. Carter, J. We plan to look at whether large errors in the P. Dallavalle, and P. D. Polger, 1989: temperature forecasts have decreased over the Evolution of the NWS national verifica- years. Finally, differences in the rates of im- tion system: Past, present, and future. provement between regions of the country Preprints 11th Conference on Probability should indicate where improvements in the and Statistics in Atmospheric Sciences, NWP models have had the greatest impact. Monterey, CA, Amer. Meteor. Soc., J41- J46. 7. ACKNOWLEDGMENTS Erickson, M. C., and J. P. Dallavalle, 1986: Over the years, numerous NWS employ- Objectively forecasting the short-range ees in NWS Headquarters, the forecast offices, maximum/minimum temperature—A new and NCEP have contributed to the establish- look. Preprints 11th Conference on ment of this verification database. We can not Weather Forecasting and Analysis, Kansas possibly name them all, but we are grateful for City, MO, Amer. Meteor. Soc., 33-38. their efforts. We thank Chad Shafer who did many of the verification diagrams shown here, Glahn, H. R., 1985: Yes, precipitation fore- Ramage, C. S., 1982: Have precipitation forecasts have improved. Bull. Amer. Meteor. casts improved? Bull. Amer. Meteor. Soc., Soc., 66, 820-830. 63, 739-743.

Glahn, H. R., and D. A. Lowry, 1972: The Ruth, D. P., and C. L. Alex, 1987: AFOS-era use of Model Output Statistics (MOS) in forecast verification. NOAA Techniques objective weather forecasting. J. Appl. Development Laboratory Computer Pro- Meteor., 11, 1203-1211. gram NWS TDL CP 87-2, 50 pp. . [Available from NWS Meteorological De- Jensenius, J. S., and M. C. Erickson, 1987: velopment Laboratory, 1325 East-West Monthly relative frequencies of precipita- Highway, Silver Spring, MD 20910-3283.] tion for the United States for 6-, 12-, and 24-h periods. NOAA Tech. Rep. NWS TDL-39, 262 pp. Zurndorfer, E. A., J. R. Bocchieri, G. M. Carter, J. P. Dallavalle, D. B. Gilhousen, Klein, W. H., and F. Lewis, 1970: Computer K. F. Hebenstreit, and D. J. Vercelli, 1979: forecasts of maximum and minimum tem- Trends in comparative verification scores peratures. J. Appl. Meteor., 9, 350-359. for guidance and local aviation/public weather forecasts. Mon. Wea. Rev., 107, Lowry, D. A., and H. R. Glahn, 1976: An op- 799-811. erational model for forecasting probability of precipitation—PEATMOS PoP. Mon. Wea. Rev., 104, 221-232.

Murphy, A. H., and T. E. Sabin, 1986: Trends in the quality of the National Weather Ser- vice forecasts. Wea. Forecasting, 1, 42- 55.