QC
NOAA Technical Memorandum ERL WMPO-29 U.S. DEPARTMENT OF COMMERCE NATIONAL OCEANIC AND ATMOSPHERIC ADMINISTRATION Environmental Research Laboratories
A STATISTICAL TECHNIQUE FOR EVALUATING HURRICANE MODIFICATION EXPERIMENTS
Richard W. Knight Glenn W. Brier
Weather Modification Program Office Boulder, Colo. May 1976 QC
'U. to wS -wo.^
NOAA Technical Memorandum ERL WMPO-29
A STATISTICAL TECHNIQUE FOR EVALUATING HURRICANE MODIFICATION EXPERIMENTS
Richard W. Knight Glenn W. Brier National Hurricane and Experimental Meteorology Laboratory Coral Gables, Florida
Weather Modification Program Office Boulder, Colo. May 1976
LIBRARY JUL 20 1976 N.QAA. U* S. Dept, of Commerce
ATMOsp^
UNITED STATES NATIONAL OCEANIC AND Environmental Research DEPARTMENT OF COMMERCE ATMOSPHERIC ADMINISTRATION Laboratories Elliot L. Richardson, Secretary Robert M White, Administrator Wilmot N Hess Director
Of °C
lb 29/6
CONTENTS Page ILLUSTRATIONS .. iv TABLES.. v ABSTRACT 1 1. INTRODUCTION 1 2. GENERAL DESCRIPTION 2 3. EXAMPLE OF THE B-SCORE COMPUTATION FROM A HYPOTHETICAL CASE 6 4. DATA REQUIREMENTS FOR THE COMPUTATION OF THE B-SCORE 9 5. SUMMARY AND CONCLUSIONS 9 6. REFERENCES 11
iii ILLUSTRATIONS
Figure Page Three forecast functions making up a forecast period 1. ℓ and having a monitoring increment period of length Δ. 3 The total monitoring period T, the forecast period 2. ℓ, and the monitoring increment number T1 to T99+m. 5
3. Pressure, temperature, and dew point observed at a hypothetical station during a month. 7 TABLES
Table Page
1. Example of matrix of observed and forecast events arranged for comparison and computation of scores 4
2. Array of indices used for B-Score computation 5
3. Change criteria and forecast function parameters used with cold frontal passages at a hypothetical 6 station
4. B-Score parameters for two hypothetical experiments 8
Y A STATISTICAL TECHNIQUE FOR EVALUATING HURRICANE MODIFICATION EXPERIMENTS
Richard W. Knight and Glenn W. Brier1
A statistical technique is developed for evaluating the non- randomized Project STORMFURY hurricane seeding experiments. Modern principles of design and analysis of comparative experiments use (1) replication, from which a quantitative estimate can be made of the experimental "error" or the variability of the response to a treatment and (2) randomization_, a process of allocating treatments to the experimental material by tossing a coin (or equivalent procedure), which may make it possible to attribute whatever effects are observed to the treatment only. Together, these two principles enable one to assess the validity of the results in terms of a probability statement. However, in the STORMFURY Project it is planned to seed nearly all experimental units, leaving essentially no controls. With the concept of randomization in time, it is possible to develop an evaluation technique to quantitatively determine whether there is an association between a treatment and the event following the treatment.
A hypothetical example is presented to facilitate the understanding of the technique; the elements that are most likely to identify seeding-induced changes in a storm environment are discussed briefly.
1. INTRODUCTION
The seeding hypothesis used in Project STORMFURY is based on definite cause and effect relationships that, in turn, are based on theoretical and observational studies. Basically, the hypothesis calls for seeding clouds at radii greater than that of the eyewall to enhance convection outside of the eyewall region. If convection can be successfully stimulated, part of the low-level inflow originally maintaining the eyewall convection will be directed into convection at a radius greater than that of the eyewall, and thus the transport of angular momentum and water vapor to the old eyewall will be reduced. As the region of major vertical mass transport is shifted to a greater radius, the maximum winds will diminish due to conservation of momentum principles. The requirements, therefore, to objectively evaluate the results of a hurricane seeding are the following: 1) Observe the entire sequence of changes that occurs in the meteorolog ical elements to test whether this sequence corresponds to the hypothesized chain of events, in space and time, and 2) Determine the statistical significance of this correspondence.
1 Glenn Brier is a Research Associate in the Department of Atmospheric Sciences, Colorado State University, Fort Collins, Colorado 80523 It is well known that the natural time and space variability of meteorological elements measured in hurricanes is very high, a fact estab lished both by land-based observations and those obtained from aircraft, (e.g., peak wind gusts may exceed 1-minute sustained wind speeds by 20 to 25 percent (Atkinson, 1974). This, of course, causes serious problems with testing the results of seeding when the classical designs for comparative experiments are used. For example, even if dramatic changes in a particu lar element were to occur after a seeding, it would take data from many trials to show that indeed there must be a seeding-induced change. The process of collecting a sufficient number of treated and control cases might take years, which is an unacceptable delay.
An alternative technique, therefore, was developed by Brier (1975) for evaluation of the results of Project STORMFURY seeding experiments. The proposed statistical technique is based on a quantitative description of the hypothesized sequence and timing of events expected to occur after a seeding treatment. It becomes obvious that the likelihood of some complex sequence occurring by chance will be considerably lower than that for a single-element event. Therefore, the technique should have the capability of providing a quantitative answer to the question: how well did the hypothetical sequence of events predict what actually happened after the seeding when compared to the variations and events during the rest of the monitoring period? Essentially, the remainder of this monitoring period acts as a control. The description of the development of the technique (hereinafter called the B-Score or B), is covered in the following section.
2. GENERAL DESCRIPTION
The objective is to compute a score or index to measure the association between the hypothesized and observed sequence of events A, B, C, ..., in a monitoring period of length T, with the forecast sequence F(A), F(B), F(C),... restricted to a period of length & when & << T. The events A, B, C,... are those defined by some physical hypothesis to follow a seeding treatment (S) when monitoring is taking place. An example would be the time required for a 1°C temperature drop in the eye of the storm. The forecasts of the events A, B, C are designated as F(A), F(B), F(C),... and specified by the hypothesis to occur after the treatment S during the interval between to and t^. This forecast time length & is defined by the time span encompassed by the distribution curves (assumed here to be Gaussian) of the forecast functions out to + 3a (fig.1). The forecast functions F(A), F(B), F(C)... are represented by normal distributions with maxima at the expected times for events A, B, C... to occur, based on the physical hypothesis. Actually, any appropriate distribution can be used for the forecast functions, and a change will be made if warranted by future experience. The standard deviations and maximum ordinates may all be different. The forecast functions are "standardized" or adjusted so that the maximum ordinates sum to unity. This provides for an index I, which has a maximum value of unity in case of perfect agreement between the forecast times ta, t^, tc and observed events at T^ and Tg, Tq respectively. The minimum value of I is zero and occurs, for example, if events A, B, C... occur outside the interval t^- tQ. Indices intermediate between zero and unity can be obtained
2 Figure 1. Three forecast functions making up a forecast period Z having a monitoring increment period of length A. The increment A is arbitrary, but should be small enough to represent the continuity in F(A) , F(B), and F(C). (Actual values of F(A), F(B), F(C), used by the computer program, are taken from the curves at points marked by •.) and the problem is to devise a score to determine whether a particular index Is (corresponding to the beginning of the sequence of events caused by seeding), is unusual with respect to the general concept of randomization in time. The following procedure is used. The hypothesized means (t^ , , tc ...) and standard deviations aa> crfc>> ac•**) of the forecast functions F(A), F(B), F(C),... respectively are specified based on previous experience and/or results of numerical hurricane models. The forecast functions all have zero value outside of the interval Z. The next step is to subtract the interval Z from the total monitoring period T, leaving a period T-£ = L (fig.2). The interval L is next divided into 99 parts so that if T^ is the beginning of the monitoring period, then
T2 = Tj + A
t3 = Ti + 2A
T100 = T + 99A
• • T = T100 + where m is an integer such that mA = Z. (This usually necessitates a slight adjustment in Z and T.) Next, the functions F(A), F(B), F(C) ... are
3 Table 1. Example* of matrix of observed and forecast events arranged for comparison and computation of scores (forecast function values are taken from points on the curves marked in fig. 1)
Observed events Forecast sequence Product values
Forecast Col 1 Monitor increment increment +Col 2 number A B c number F(A) F(B) F(C) AxF(A) Bx(B) CxF(C) +Col
i 0 0 0 1 .00 .00 .00 0 0 0 0 2 0 0 0 2 . A2 .01 .00 0 0 0 0 3 1 1 0 3 .13 .15 .00 .13 .15 0 .28 A 0 0 1 A .00 .31 .03 0 0 .03 .03 5 0 0 0 5 .00 .09 .08 0 0 0 0 6 0 0 0 6 .00 .00 • 1A. 0 0 0 0 7 0 1 0 7 .00 .00 .15 0 0 0 0 8 0 1 1 8 .00 .00 .11 0 0 .11 .11 9 0 0 0 9 .00 .00 .0A 0 0 0 0 10 0 0 0 10 .00 .00 .01 0 0 0 0 11 0 0 0 12 1 0 0
99 0 0 0
99+M 0 0 0 .42
* This is the first of 100 tables which are used to compute 100 indices 1^. computed at locations tlf...t^ on the segment £ where
t + A 2
t 3 + 2
t = t-, + mA.
Table 1 is then prepared where the columns A, B, C represent the observed events, zero indicates the nonoccurence of the event, and unity represents the occurrence. An event may occur more than once during the monitoring period. For example, event C could be a decrease of wind speed greater than 10 kt and could occur at time T4 and Tg. The next three columns show the forecast
4 H— R~r r T, T,T,T4 M+m
t,—<-t Figure 2. The total monitoring period T, the forecast period l, and the mon itoring increment numbers Ti to Tgg+m are depicted. Arrows at bottom of figure show how the forecast period £ is moved along the monitoring period T to generate indicies Ij_.
sequences positioned so that t1 is matched up with Tj (fig.2). The numbers in these columns are values taken from the normal curve of each respective forecast function at m intervals of A minutes (marked in fig-1). The next three columns are scores derived from the previous columns by matching the observations with the corresponding forecast as indicated. The last column is the sum of AxF(A), BxF(B), ...) and the entry in the last line of the table is obtained by summing the last column. This is one of the 100 indices Ij^ that are to be computed. The values in each line of the forecast sequence in table 1 can now be displaced one line downward so that tj is now matched with Tg, t^ is matched with T , etc. The scores are computed as before, and these steps are continued until t is lined up with Tlnf). This exhausts all the comparisons, numbering 100 in1 all.
Table 2 is a summary showing the scores obtained for each of the 100 possible positions. The index I represents the total score corresponding to the time when tj is matched up to the real time Ts, specified by the seeding hypothesis.
Now all 100 values of the index I are compared to the index Is that was
Table 2. Array of indices used for B-Score computation
Position of i± relative to ti i ii 2 T-2 3 I3 4 14 5 15 • •
• • s Is • • • • 99 100 t99 noo 5 Table 3. Change criteria and forecast function parameters used with cold frontal passages at a hypothetical station Times of occurrence after a significant cold frontal passage (hours)
Measured ElementCriterionMeanS.D. A Temperature Fall of 15°F 9 3 B Dewpoint Fall of 20°F 20 5 C PressureRise of 4 mb40 10 obtained at the time of seeding Ts. The B-score is obtained by counting the number of indices that equal or exceed Is. The B-score, therefore, takes on -values between 1 and 100 and is used to determine whether there is an un usual correspondence between the actual events, A, B, C, ••• and the predict ions, F(A), F(B), F(C), ... of the seeding hypothesis. For example,a B-score of 50 would suggest no significant relationship, since one would expect by chance that 50 percent of the scores would be higher (or lower) than Ig when no association existed between the forecast and observed events. Thus, the B—score can be converted into a probability by division by 100 to indicate whether the null hypothesis could be rejected at some chosen confidence level. Now a judgment can be made whether there was significant association between the actual events and those specified by the seeding hypothesis. Of course, this is based on the assumption that the probability of the sequence of events A, B, C... is uniform over the interval L and that the time of seeding Tg is randomly chosen in this interval. Otherwise, the accusation might be made that the experimenter had forecasting ability and introduced a bias by selecting a more favorable time for application of the treatment•
3. EXAMPLE OF THE B-SCORE COMPUTATION FROM A HYPOTHETICAL CASE
In this section, the use of the B-Score is demonstrated by applying it to a hypothetical weather situation. Two cold fronts have passed a station during a month. The score is used to test whether either of these fronts had significantly affected the observed weather after their passage.
Three elements are chosen for monitoring that can be easily measured and that can provide a good indication that the event (cold frontal passage) changed the weather as hypothesized. The three elements are the temperature, dew point, and pressure measured at the station. A 1—month record of these is shown in figure 3. Table 3 lists the change criteria chosen as indicators of significant weather change and the appropriate forecast functions. In this situation, the forecast functions were chosen by looking at past observations of frontal passages at the station, and then basing the forecast functions on this prior record of observations. Together, the three change- criteria make up a sequence of events tnat^ when observed, indicates the occurrence of the hypothesized effect, or in this case, a significant change in the weather caused by a cold frontal passage.
6 15 30 1 DAYS
Figure 3. Pressure, temperature, and- dew point observed at a hypothetical station during a month. The times when the chosen criteria are satis fied are marked with numbers. Heavy vertical lines indicate frontal passages on the 10th and 20th at 002.
After choosing appropriate forecast functions, the total observation period is divided into two monitoring periods so that the cold frontal passages occurring on the 10th and 20th of the month can be dealt with inde pendently. We do this so that the test of significance of one frontal passage will not be in any way obscured by the effects of another. The next step is to compute the B-Score parameters, that is, the values for the monitoring period T, the forecast function period £, the monitoring interval A and the number of increments M (table 4). As noted in section 2, slight adjustments usually have to be made in T and £ so that M and A are integers. The number of integers computed for comparison purposes does not have to be 100; adjust ments can be made if the data require it. There are two reasons for making this adjustment. First, if one or more of the forecast functions are relative- y sharp, an adjustment may be made to obtain adequate resolution of the particular function. Figure 1 can be used to illustrate this point. In the figure, the curve F(A) is relatively sharp. Since MA =■ £, as M becomes smaller A becomes larger, which could result in F(A) being missed, or being poorly represented in the computer program. Another reason for changing A may be convenience, that is, to make it conform to the measuring increment of the observational data. From the definitions of the B-Score parameters, the following equation can be derived that shows the relationship between the M and the number of indices:
M = ^ ^ (no. of indices) - 1 .
7 Table 4. B-Score parameters for two hypothetical experiments
Experiment i 2 Original monitoring period 360 hours 384 hours Original forecast function period 70 hours 70 hours Adjusted monitoring period (T) 366 hours 366 hours Adjusted forecast function period (£) 69 hours 69 hours Monitoring interval (A) 3 hours 3 hours Number of increments (M) 23 hours 23 hours
Since T and £ are constants (both may be adjusted slightly so that M is an integer) for a given experiment, M is proportional to the number of comparisons made.
For the first experiment, the original monitoring period was 15 days or 360 hours. This was adjusted to 366 hours or 15.3 days. The forecast function period was adjusted from 70 to 69 hours. Similar adjustments were made to the second experiment (table 4). Next, the observed data are checked for changes that meet the chosen criteria (fig. 3). These times are then converted into a corresponding increment number ranging from 1 to 99+M. (For the general case, increment numbers range from 1 to A+M where A is some reasonably large number; usually _> 25). Similarly, the times of the cold frontal passages are converted, and this information is loaded into the computer program, which computes the B-score for each time that a frontal passage occurred. The probabilities obtained are then used to accept or reject the null hypothesis.
In this case, the null hypothesis HQ states that the cold frontal passages at the station produced no significant change in the subsequent observed weather. Thus, any observed changes could have been well within the bounds of the natural variations that are observed when there are no frontal passages. The B-Score probabilities are used to reject HQ at a chosen level of significance.
The observational data occurring on the 10th show excellent correspond ence with the hypothesized sequence of events that is expected to occur after a significant cold front (fig. 3). The temperature and dew point dropped rapidly through the chosen criteria, and the pressure rose slowly through its threshold value as required by the hypothesized sequence of events. On the other hand, the events occurring after the cold front passed on the 20th show poorer correspondence with the hypothesis. The pressure rose simultaneously with the frontal passage. The temperature dropped as required, but was considerably late, and the dew point criterion was not met at all. The B-Score probabilities obtained for the two cold fronts were .03 and .08 respectively. Therefore, in the case of the cold front on the 10th, H0 can be rejected ait the 3 percent level, which is strong evidence that indeed there was a significant change in the weather, as we have defined it, after the frontal passage. In the second case, an examination of the raw data appears to indi cate a frontal effect. However, based on the data observed during the rest of the month, there is one chance in 12 that the observed sequence of events could 8 have occurred with no frontal passage. Therefore, in the second case H0 would probably be accepted.
The above example demonstrates the ability of the technique to assess statistical significance of the relationship between forecast and observed phenomena that have a large degree of natural variability. Had the forecast functions been used separately, their power to reject the null hypothesis would have been minimal because each change-criterion was met one or more times when there was no frontal passage. However, when combined into a sequence of events the method becomes a powerful tool for evaluating hypothesized cause-and-effect relationships.
4. DATA REQUIREMENTS FOR COMPUTATION OF THE B-SCORE
With respect to a hurricane seeding experiment, the next task is to identify the elements which will be used in the computation of Bg. At this time, it is not known precisely which of these will provide the best evidence of a seeding effect. However, a wealth of meteorological data will be available for analysis as a result of measurements made by NOAA aircraft instruments that will monitor seeded hurricanes. In addition, the variation al optimization approach (Sheets, 1973) may be used to filter the data so that the contributions by the various scales of motion within the storm may be scrutinized.
The data and the analysis techniques will be available; the problem is to choose the appropriate data for the forecast functions, that is, the elements that provide the best evidence of the occurrence or non-occurrence of a seeding effect. The results of research using airborne measurements made during the Florida Area Cumulus Experiment (FACE 1975) should provide some valuable insight into this problem for response of individual clouds.
Seeding experiments on Hurricane Debbie in 1969 and research conducted during Project STORMFURY cloudline exercises (Sheets and Pearce 1975) indi cate moisture distribution changes as well as changes in the temperature and pressure patterns after seeding. Of course, changes in wind are expected also. Initially, therefore, the forecast functions for the computation of Bs will be based on these four elements together, or in various combinations. Additionally, radar echo signatures may prove to be useful indicators of seeding-induced changes.
5. SUMMARY AND CONCLUSIONS
If hurricanes can be successfully modified, the resultant changes are expected to be similar in magnitude to naturally occurring changes. This makes it somewhat difficult to obtain convincing evidence of seeding-induced changes using standard statistical methods. Therefore, the evaluation of hurricane modification experiments has and continues to rely on cause and effect relationships. The technique described in this paper is based on the sequence and timing of more than one event and should provide a powerful statistical analysis tool which will greatly supplement the physical analysis approach. It gives a quantitative assessment as to whether a specified series of events occurred as hypothesized.
Currently, there is little intuition about what the forecast functions should be. However, after continuous long term monitoring 30 hours) of specified parameters is accomplished in a few more seeded and unseeded storms, we should have a much better idea about which of the meteorological elements provide evidence of seeding induced change and what their respective forecast functions should he. With this information, the B-Score technique should prove to be an excellent means of attaching statistical significance to the results of hurricane seeding experiments.
10 6. REFERENCES
Atkinson, G. D., 1974: FLEWEACEN/Joint Typhoon Warning Center Tech. Note 74-1, Guam.
Brier, G. W., 1975: Statistical design and evaluation of the FACE and STORMFURY programs. Interim report for NOAA-NHEML, U.S. Department of Commerce, NOAA, Coral Gables, Florida.
Panofsky, H. A. and G. Brier, 1958: Some Applications of Statistics to Meteorology. University Park, The Pennsylvania State University, 224 pp.
Sheets, R. C. and S. C. Pearce, 1975: A case study of two STORMFURY tloudline seeding events. NOAA Tech. Memo. ERL WMPO-21, February, 45 pp.
11 USCOMM - ERL