Characterization of Background Water Quality for Streams and Groundwater

NATURAL RESOURCES DEFENSE COUNCIL’S & POWDER RIVER BASIN RESOURCE COUNCIL’S PETITION FOR REVIEW

EXHIBIT 14

5644

CHARACTERIZATION OF BACKGROUND WATER QUALITY FOR STREAMS AND GROUNDWATER

FERNALD ENVIRONMENTAL MANAGEMENT PROJECT FERNALD, OHIO

REMEDIAL INVESTIGATION and FEASIBILITY STUDY

I-

May 1994 .

U.S. DEPARTMENT OF ENERGY FERNALD FIELD OFFICE

DRAFT’”FINAL 4 .,. ., J I 5644 Background Study May 1994

TABLE OF CONTENTS

Eiw? List of Tables iv List of Figures vii List of Acronyms viii 1.0 Introduction - 1-1 1.1 Brief History of the Site 1-1 1.2 Purpose 1-3 1.3 Geologic Setting 1-4 1.4 Hydrologic Setting 1-7 1.4.1 Great Miami River 1-7 1.4.2 Paddys Run 1-10 1.4.3 Great Miami Aquifer 1-11 1.4.4 Glacial Overburden 1-13 1.4.5 Monitoring Wells 1-13 1.5 EPA Guidance on Background Characterization 1-15 1.6 Summary of Revisions Made to the Draft Report (May 1993) 1-16 2.0 Previous Studies 2-1 2.1 Environmental Monitoring Program 2-1 2.2 U.S. Geological Survey Surface Water Monitoring 2-1 2.3 U.S. Geological Survey Groundwater Study 2-3 2.4 IT Corporation Final Interim Report 2-3 2.5 Argonne National Laboratory Environmental Survey 2-3 2.6 Ohio Department of Health 2-3 2.7 Ohio Environmental Protection Agency Study of the Great Miami River 2-6 2.8 Previous RI/FS Background Studies 2-6 2.9 RCRA Groundwater Monitoring 2-7 3.0 Development of the RI/FS Background Data Set 3-1 3.1 Sampling Locations 3-1 3.1.1 Surface Water, 3-1 3.1.2 Groundwater 3-2 3.1.2.1 Identification of Potential Background Monitoring Wells 3-2 3.1.2.2 Screening of Background Locations 3-4 3.1.2.3 General Water Chemistry and Charge Balance 3-11 3.1.2.4 Summary 3-15 3.2 Sample Collection 3-19 3.3 Analytical Procedures 3-20

0USIG:BACKGRDISECS.1-9.TOC.IOS-W i Background Study May 1994

TABLE OF CONTENTS (continued) Page

3.4 Data Validation Procedures 3-20 3.5 Validated and Deleted Data 3-21 4.0 Data Set Modifications and Statistical Analysis Procedures 4-1 4.1 Overview 4-1 4.2 Modifications of the Background Data Set 4-1 4.2.1 Treatment of Rejected/Nonvalidated Data 4-2 4.2.2 Treatment of Nondetect Data 4-2 4.2.3 Identification and Treatment of Outliers and Other "Suspect" Data 4-3, 4.2.4 Data Averaging 4-5 4.3 Statistical Analysis 4-5 4.3.1 Testing of Data Distribution 4-5 4.3.2 Parametric Descriptive Statistics 4-7 4.3.3 Nonparametric Descriptive Statistics 4-7 4.3.4 Comparison of Populations 4-8 4.4 Summary of Revisions to Chapter 4 of the Draft Report 4-8 5.0 Glacial Overburden 5-1 a 5.1 Radiological Constituents 5-1 5.2 Inorganic Constituents 5-2 5.3 Organic Constituents 5-3 6.0 Great Miami Aquifer 6-1 6.1 Radiological Constituents 6-1 6.2 Inorganic Constituents 6-2 6.3 Organic Constituents 6-3 7.0 Great Miami River 7-1 7.1 Radiological Constituents 7-1 7.2 Inorganic Constituents 7-1 7.3 Organic Constituents 7-2 8.0 Paddys Run 8-1 8.1 Radiological Constituents 8-1 8.2 Inorganic Constituents 8-1 8.3 Organic Constituents 8-2 9.0 Conclusions 9-1 References R-1.

0USIG:BACKGRDISECS.1-9.,W./OS-W ii 5644.- Background Study May 1994

TABLE OF CONTENTS (continued)

Appendix A - Data from Previous Studies Appendix B - Drilling Logs and Well Construction Information Appendix C - Radiological Data Appendix D - Inorganic Chemical Data Appendix E - Organic Chemical Data Appendix F - Statistical Procedures, Equations, and Results Appendix G - Summary of Revisions to the "Characterization of Background Water Quality for Streams and Groundwater" Draft Report (May 1993) Appendix H - Summary Statistics of Inorganic Constituents for Background Monitoring Wells in the Tributary Sections of the Great Miami Aquifer

000004

OU5IG:BACKGRDISECS.1-9.TOc./OS-W iii Table E-19 Rejected/Nonvalidated Organic Data for Background Surface Water in the Great Miami River

Well Sample lab Validated . QA No. Date ID qualifier Constituent Result Qualifier type W-1 05/20/93 120064-2 U 4-Nitroaniline 25 R N W-1 05/20/93 120068-1 U 4-Nitroaniline 25 R D W-1 05/20/93 120072-2 U 4-Nitroaniline 25 R T

.. ., ... E-210 c'

Table E-20 Rejected/Nonvalidated Organic Data for Background Surface Water in Paddys Run

Well Sample lab Validated QA No. Date ID qualifier Constituent Result Qualifier type W-5 03/25/93 113493 U 2,4-Dinitrophenol 50 R N

I W-5 03/25/93 113493 U 4,6-Dinitro-2-methylphenol 25 R N

E-21 1 d.

APPENDIX F STATISTICAL PROCEDURES, EQUATIONS, AND RESULTS TABLE OF CONTENTS

List of Tables F-ii Shapiro-W& Test for Normality F-5 Shapiro-Francia Test for Normality F-14 Rosner's Test for Many Outliers F-23 Data Averaging F-30 Sample Arithmetic Mean - Normal Distriiution F-33 Sample Arithmetic Standard deviation - Normal Distribution F-34 Estimated Coefficient of Variation - Normal Distribution F-35 Estimated Mean of a Lognormal Distribution F-36 Estimated Standard Deviation of a Lognormal Distribution F-38 Sample Median - Nonparametric Technique F-39 Upper One-Sided 95% Confidence Limit - Normal Distribution F-40 Upper One-sided 95% Confidence Limit - Lognormal Distribution F-43 Upper One-sided 95% Confidence Limit - Nonparametric Technique F-47 95'h Percentile - Normal Distribution F-49 9SthPercentile - Lognormal Distribution F-50 95'h Percentile - Nonparametric Technique F-51 F-Test F-52 T-Test F-57 The Wilcoxin Rank Sum Test F-60 Kruskal-Wallis Test F-64

F-i 000924 LIST OF TABLES

F- 1 Formulas for Summary Statistics F- 1 F-2 Coefficients 3 for the Shapiro-Wilk W Test for Normality F-7 F-3 Quantiles of the Shapiro-Wilk W Test for Normality (Values of W Such that 100 p% of the Distribution of W Is Less Than WJ F-9 F-4 Example Data Set Number 1 F-11

F-5 Standard Normal Curve for a Z Distribution F-16 F-6 Percentage Points of the W' Test for n > 50 F-18 F-7 Example Data Set Number 2 F-2 1

F-8 Approximate Critical Values Lamda (it 1 ) for Rosner's Generalized ESD Many-Outlier Procedure for alpha = 0.05 F-25 F-9 Example Data Set Number 3 F-27 F-10 Example Data Set Number 4 F-3 1 F-11 Quantiles of the t Distribution (Values oft Such that 100 p% of the Distribution Is Less Than 5) F-4 1 F-12 Values of H (1-alpha) for Computing One-sided (Upper) 95% Confidence Limits on a Lognormal Mean F-44 F-13 Percentage Points of the F Distribution (Fo.025,dm,dn) F-53 F-14 Example Data Set Number 5 F-55 F-15 Example Data Set Number 6 F-62 F-16 Quantities of the Chi-square Distribution With v Degrees of Freedom F-67 F-17 Ranking of Example Data Set Number 7 F-68 F-18 Ranking of Example Data Set Number 7 By Group F-69

F-ii

000925 FEMP Background Study . May 1 Table F-1 Formulas for Summary Statistics

statistic Formula

ShapiiW& Test (Gilbert 1987, Equations 12.3 and 12.4) 2

where: n n2

d E C xi2 -- i=l f,Pi=l 4

n k =; ifnisevcn -n-1 = ifnisodd a, = Shapiro-Wilk coefficient x, = ith data value in the ordered data set = square of the ith data value in the ordered data set n = numberofdatapoints W = Shapim-Wilk test statistic

ShaphFrancia Test (Shapiro-Francia, 19tz)

where:

mi = normal qwntile a-1 = inverse of standard normal distribution x, = ith data value in the ordered data set s2 = sample arithmetic variance W' = Shapiro-Francia test statistic

__~ Rosner's Test for Many Outliers (Gilbert 1987, See page F-23. Equations 15.1 to 15.3)

Sample Arithmetic Mean (Gilbert 1987, Equation 43) .'--cyIn i.1

where: n = number of data points -x, = i* data value in the ordered data set x = arithmetic mean

F- 1 FEMP Background Study 64% May 1994

TABLE F-1 (Continued)

Statistic Formula

Sample Arithmetic Standard Deviation (Gilbert 1987, Equation 4.4)

whcm: n = number of data points 3 = data Set value x = arithmetic mean s2 = arithmetic variance s = arithmetic standard deviation

Estimated Coefficient of Variation (Gilbert 1987, Page 34) CV - sfi

where: -x = sample arithmetic mean s = sample arithmetic standard deviation CV = estimated coefficient of variation

Estimated Mean of a Lognormal Distribution (Gilbert 1987, Equation 13.7) ji - exp [. + $1

where:- y = arithmetic mean of the In transformed data 6y = arithmetic standard deviation of the In transformed data

A p = estimated mean of a lognormal distribution

Estimated Standard Deviation of a Lognormal Distribution (Gilbert Equation 1987, 13.8) b - /b' [ exp s; - 11

where:

A = estimated mean of a lognormal distribution P 6y = arithmetic standard deviation of the In transformed data = estimated standard deviation of the lognormal distribution

000927

F-2

*.n r.'.., -...3...'2 TABLE F-1 (Continued)

statistic Formula Simple Median - Nonparametric Technique Ifnisodd (Gilbert 1987, Equation 13.15 and 13.16) sample median = xI(" .

If n is cvcn:

whCX xm = i" data value in the ordered data set n = number of data points

~ ~~~ Upper 95% Confidence Limit on the Mean - Normal Distribution (Gilbert 1987, Equation 11.6)

whem- x = arithmetic mean t o.95,n-, = student t distribution value n = number of data points s = arithmetic standard deviation 95% UCL, = one-sided upper 95% confidence limit for a normal distribution

Upper 95% Confidence Limit on the Arithmetic Mean for Lognormal Distribution (Gilbert 1987, Equation 13.13)

where:- y = arithmetic mean of the In transformed data sy2 = arithmetic variance of the In transformed data sy = arithmetic standard deviation of the In transformed data b.95= value used to compute one-sided 95% confidence limit on a lognormal mean n = number of data points 95% Uq= one-sided upper 95% confidence limit for a lognormal distribution

.. 000928 F-3 PeMP Background Study May 1994

TABLE F-1 (Continued)

statistic Formula Upper 95% Confidence Limit on the Median - Nonparametric Technique (Gilbert 1987, Equation 1322)

wbelC: n = number of data points %.95 = upper 95% limit from a standard normal curve for a Z distribution U = rank in an ascending order data set that corresponds to the one-sided upper 95% confidence limit for nonparametric distribution f(U) = U rounded up to an integer (e.&, 242 -. 25) *. 95% UCL,.,, = data point in the ascending ordered data set at rank f(u)

0-95 Quantitle - Nonnal Distribution (Gilbert 1987. Equation 11.1)

where: -x = arithmetic mean s = arithmetic standard deviation &,g5 = 0.95 limit from a standard normal curve for a Z distribution 95" PercentileN = 9Sth percentile for normal distribution

0.95 Quantile - Lognormal Distribution (Gilbert 1987, Equation 13.24)

where: 7 = arithmetic mean of the In transformed data 5 = arithmetic standard deviation of the In transformed data Z& = upper 95% limit from a standard normal curve for a distribution 9SthPercentileL = 9Sth percentile for lognormal distribution

0.95 Quantilc - Nonparametric Technique Q = 0.95 n sthPercentileNp - x~~(~)~

where: n = number of data points Q'= rank in an ascending order data set that corresponds to the 95" percentile based on a nonparametric technique

f(Q) = Q rounded up to an integer (e.& 14.1 4 1s) 95" PercentileNp = data point in the ascending order data bet at rank f(Q)

Ptat See WEC F-52.

T-tat See pa@ F-57.

Wilcoxon Rank Sum tat Set page F-60. hkal-Wallis test See page F-64.

F-4 (900929 FEW Background Study May 1994 Shapiro-Wilk Test for Normality

The W test developed by Shapiro and Wilk (Gilbert 1987, Equations 12.3 and 12.4) was used to determine whether or not a data set has been drawn from a population which is normally distributed for sample size of 50 or less. By conducting this test on the natural logarithm of each data value, the W test was used to determine whether or not the sample was drawn from an underlying lognormal distribution. The null hypothesis to be tested is:

The population has a normal (lognormal when the data is transformed) distribution. versus

HA: The population does not have a normal (lognormal when the data is transformed) distribution.

If H, is rejected, then HA is accepted.

The following presents a step-by-step procedure for conducting the W test. The equation for calculating W is: k l2 1. Compute the denominator (d) of the W test statistic

2 n n i=l

F-5 000930 FEW Background Study May 1994 where: n c x, = x, + x2 + .... + xn i=l

n 2 2 c 4 =xi2 +x2 + ... +d i=l

2. Order the n data points in ascending order (smallest to largest) such that x, - x2 5 x3 I...< q 3. Compute k, where: n k = - ifn is even 2

n-1 k = -ifn is odd 2

4. Find the coefficients a,, +, a,, ..., ak for the sample size n from Table F-2.

5. Compute W

w=-1 d

6. Reject H,,at the a significance level if W is less than the quantile given in Table F-3. To test the null hypothesis that the population has a lognormal distribution, transform the observed data to y,, y2, ..., yn where yi = In 3. Repeat steps 1 through 6 as described above.

F-6

000931 I i

- May 1994

3 4 5 6 7 8 9 10 0.7071 0.6872 0.6646 0.6431 0.6233 0.6052 0.5888 0.5739 O.OOO0 0.1677 0.2413 0.2806 0.3031 0.3164 0.3244 0.3291 0.0000 0.0875 0.1401 0.1743 0.1976 0.2141 4 O.oo00 0.0561 0.0947 0.1224 - 0.0000 0.0399

11 12 13 14 15 16 19 20 E1 0.5601 0.5475 0.5359 0.5251 0.51 50 0.5056 0.4808 0.4734 2 0.331 5 0.3325 0.3325 0.3318 0.3306 0.3290 0.3232 0.321 1 3 0.2260 0.2347 0.2412 0.2460 0.2495 0.2521 0.2561 0.2565 4 0.1429 0.1586 0.1707 0.1802 0.1878 0.1939 0.2059 0.2085 '5 0.0695 1 0.0922 0.1099 0.1240 0.1353 0.1447 0.1641 0.1686 6 0.0000 0.0303 0.0539 0.0727 0.0880 0.1005 0.1271 0.1334 7 0.0000 0.0240 0.0433 0.0593 0.0932 0.1013 0.0000 0.0196 0.061 2 0.071 1 0.0303 0.0422 I 0.0000 0.0140

~ ~ 24 25 26 27 28 29 30 E1 0.4493 0.4450 0.4407 0.4366 0.4328 0.4291 0.4254 2 0.3185 0.3156 0.3126 0.3098 0.3069 0.3043 0.3018 0.2992 0.2968 0.2944 3 0.2578 0.2571 0.2563 0.2554 0.2543 0.2533 0.2522 0.2510 0.2499 0.2487 4 0.21 19 0.2131 0.2139 0.2145 0.2148 0.21 51 0.21 52 0.2151 0.2150 0.2148 5 0.1736 0.1764 0.1787 0.1807 0.1822 0.1836 0.1848 0.1857 0.1864 0.1870 6 0.1399 0.1443 0.1480 0.1512 0.1539 0.1563 0.1584 0.1601 0.1616 0.1630 7 0.1092 0.1150 0.1201 0.1245 0.1283 0.1316 0.1346 0.1372 0.1395 0.1415 8 0.0804 0.0878 0.0941 0.0997 0.1046 0.1089 0.1 128 0.1162 0.1192 0.1219 9 0.0530 0.0618 0.0696 0.0764 0.0823 0.0876 0.0923 0.0965 0.1002 0.1036 10 0.0263 0.0368 0.0459 0.0539 0.0610 0.0672 0.0728 0.0778 0.0822 0.0862 11 0.0000 0.0122 0.0228 0.0321 0.0403 0.0476 0.0540 0.0598 0.0650 0.0697 12 0.0000 0.0107 0.0200 0.0284 0.0358 0.0424 0.0483 0.0537 13 0.0000 0.0094 0.01 78 0.0253 0.0320 0.0381 14 0.0000 0.0084 0.0159 0.0227 -15 0.0000 0.0076

000932 F-7 FEMP Background Study May 1994

Table F-2 (Continued) Coefficients ai for the ShapireWilk W Test for Normality E 31 32 33 -34 35 36 37 38 39 40 1 0.4220 0.41 88 0.4156 0.4127 0.4096 0.4068 0.4040 0.401 5 0.3989 0.3964 2 0.2921 0.2898 0.2876 0.2854 0.2834 0.2813 0.2794 0.2774 0.2755 0.2737 3 0.2475 0.2462 0.2451 0.2439 0.2427 0.2415 0.2403 0.2391 0.2380 0.2368 4 0.2145 0.2141 0.21 37 0.2132 0.21 27 0.2121 0.21 16 0.21 10 0.21 04 0.2098 5 0.1 874 0.1878 0.1880 0.1882 0.1883 0.1883 0.1883 0.1881 0.1880 0.1878 6 0.1641 0.1651 0.1660 0.1667 0.1673 0.1678 0.1683 0.1686 0.1689 0.1691 7 0.1433 0.1449 0.1463 0.1475 0.1487 0.1496 0.1505 0.1513 0.1520 0.1526 8 0.1243 0.1265 0.1284 0.1301 0.1317 0.1331 0.1344 0.1356 0.1366 0.1376 9 0.1066 0.1093 0.1 116 0.1140 0.1 160 0.1179 0.1 196 0.121 1 0.1225 0.1237 10 0.0899 0.0931 0.0961 0.0988 0.1013 0.1036 0.1056 0.1075 0.1092 0.1 108 11 0.0739 0.0777 0.081 2 0.0844 0.0873 0.0900 0.0924 0.0947 0.0967 0.0986 12 0.0585 0.0629 0.0669 0.0706 0.0739 0.0770 0.0798 0.0824 0.0848 0.0870 13 0.0435 0.0485 0.0530 0.0572 0.061 0 0.0645 0.0677 0.0706 0.0733 0.0759 14 0.0289 0.0344 0.0395 0.0441 0.0484 0.0523 0.0559 0.0592 0.0622 0.0651 15 0.0144 0.0206 0.0262 0.0314 0.0361 0.0404 0.0444 0.0481 0.051 5 0.0546 16 0.0000 0.0068 0.01 31 0.0187 0.0239 0.0287 0.0331 0.0372 0.4090 0.0444 17 0.0000 0.0062 0.01 19 0.01 72 0.0220 0.0264 0.0305 0.0343 18 0.0000 0.0057 0.01 10 0.0158 0.0203 0.0244 19 0.0000 0.0053 0.0101 0.0146 -20 0.0000 0.0049 - Ei 41 42 43 44 45 46 47 -48 49 50 1 0.3940 0.391 7 0.3894 0.3872 0.3850 0.3830 0.3808 0.3789 0.3770 0.3751 2 0.271 9 0.2701 0.2684 0.2667 0.2651 0.2635 0.2620 0.2604 0.2589 0.2574 3 0.2357 0.2345 0.2334 0.2323 0.2313 0.2302 0.2291 0.2281 0.2271 0.2260 4 0.2091 0.2085 0.2078 0.2072 0.2065 0.2058 0.2052 0.2045 0.2038 0.2032 5 0.1876 0.1874 0.1871 0.1868 0.1865 0.1862 0.1859 0.1855 0.1851 0.1847 6 0.1693 0.1694 0.1695 0.1695 0.1695 0.1695 0.1695 0.1693 0.1692 0.1691 7 0.1531 0.1535 0.1539 0.1542 0.1545 0.1548 0.1550 0.1551 0.1553 0.1554 8 0.1384 0.1392 0.1398 0.1405 0.1410 0.1415 0.1420 0.1423 0.1427 0.1430 9 0.1249 0.1259 0.1269 0.1278 0.1286 0.1293 0.1300 0.1306 0.1312 0.1317 10 0.1 123 0.1 136 0.1149 0.1 160 0.1170 0.1 180 0.1 189 0.1 197 0.1205 0.1212 11 0.1004 0.1020 0.1035 0.1049 0.1062 0.1073 0.1085 0.1095 0.1105 0.1 113 12 0.0891 0.0909 0.0927 0.0943 0.0959 0.0972 0.0986 0.0998 0.1010 0.1020 13 0.0782 0.0804 0.0824 0.0842 0.0860 0.0876 0.0892 0.0906 0.091 9 0.0932 14 0.0677 0.0701 0.0724 0.0745 0.0765 0.0783 0.0801 0.081 7 0.0832 0.0846 15 0.0575 0.0602 0.0628 0.0651 0.0673 0.0694 0.071 3 0.0731 0.0748 0.0764 16 0.0476 0.0506 0.0534 0.0560 0.0584 0.0607 0.0628 0.0648 0.0667 0.0685 17 0.0379 0.041 1 0.0442 0.0471 0.0497 0.0522 0.0546 0.0568 0.0588 0.0608 18 0.0283 0.031 8 0.0352 0.0383 0.041 2 0.0439 0.0465 0.0489 0.051 1 0.0532 19 0.0188 0.0227 0.0263 0.0296 0.0328 0.0357 0.0385 0.041 1 0.0436 0.0459 20 0.0094 0.0136 0.01 75 0.021 1 0.0245 0.0277 0.0307 0.0335 0.0361 0.0386 21 0.0000 0.0045 0.0087 0.01 26 0.0163 0.0197 0.0229 0.0259 0.0288 0.031 4 22 0.0000 0.0042 0.0081 0.01 18 0.0153 0.0185 0.0215 0.0244 23 0.0000 0.0039 0.0076 0.01 11 0.0143 0.01 74 24 0.0000 0.0037 0.0071 0.01 04 -25 0.0000 0.0035 source: Table A-6, Gilbert 1987.

*/,.. I.., , Ui' u ' ' CIN/OU5RI /419195/TABF2.XLS/5-94 F-8 000935 f 5644

FEMp Background Study May 1994 Table F-3 Quantiles of the Shapiro-Wdk W Test for Normality (Values of W Such That 100 p% of the Distribution of W Is Less Than W,,)

n wo01 wo0.2 Wo 05 wo woso 3 0.753 0.7% 0.767 0.789 0.959 4 0.687 0.707 0.748 0.792 0.935

5 ' -. 0.686 0.715 0.762 0.806 0.927 6 0.713 0.743 0.788 0.826 0.927 7 0.m 0.760 0.803 0.838 0.928 8 0.749 0.78 0.818 0.851 0.932 9 0.764 0.791 0.829 0.859 0.935 10 0.781 0.806 0.842 0.869 0.938 11 0.792 0.817 0.850 0.876 0.940 12 0.805 0.828 0.859 0.883 0.943 13 0.814 0.837 0.866 0.889 0.945 14 0.825 0.846 0.874 0.895 0.947 1s 0.835 OS5 0.881 0.901 0.950 16 0.844 0.863 0.887 0.906 0.952 17 OS1 0.869 0.892 0.910 0.954 18 0.858 0.874 0.897 0.914 0.956 19 0.863 0.879 0.901 0.917 0.957 20 0.868 0.884 , 0.905 0.920 0.959 21 0.873 0.888 0.908 0.923 0.960 22 0.878 0.892 0.911 0.926 0.961 23 0.881 0.8% 0.914 0.928 0.%2 24 0.884 0.898 0.916 0.930 0.963 25 0.886 0.901 0.918 0.931 0.964 24 0.891 0.904 0.920 0.933 0.965 27 0.894 0.906 0.923 0.935 0.965 28 0.8% 0.908 0.924 0.936 0.966 29 0.898 0.910 0.926 0.937 0.966 30 0.900 0.912 0.927 0.939 0.967 31 0.902 0.914 0.929 0.940 0.967 32 0.904 0.91s 0.930 0.941 0.968 33 0.906 0.917 0.931 0.942 0.968 34 0.908 0.919 0.933 0.943 0.969 35 0.910 0.920 0.934 0.944 0.969 36 0.912 0.922 0.935 0.945 - 0.970 37 0.914 0.924 0.936 0.946 0.970 38 0.916 0.925 0.938 0.947 0.971 3? 0.917 0.927 0.939 0.948 0.971 40 0.919 0.928 0.940 0.949 0.971 41 0.920 0.929 0.941 0.950 0.972 42 0.922 0.930 0.942 0.951 0.972 43 0.923 0.932 0.943 0.951 0.973 44 0.924 0.63 0.944 0.952 0.973 45 0.926 0.934 0.945 0.953 0.973 46 0.927 0.935 0.945 0.953 0.974 47 0.928 0.936 0.946 0.954 0.974 48 0.929 0.937 0.947 0.954 0.974 49 0.929 0.937 0.947 0.955 0.974 50 0.930 0.938 0.947 0.955 0.974

Source: Table A-7, Gilbcrt 1987

F-9 000934

. , '.. i. ', 1 FEMP Background Study May 1994 Example:

To illustrate the application of the Shapiro-Wilk test, the data in Table F-4 is used.

1. Compute d, the denominator of the W test.

2 n n i=l

1 = 0.0808 - - (1.634)* 36

= 0.0066

2. Order data points from low to high.

3. Compute k for n = 36

4. Find coefficients a,, a,, ..., aU for n = 36 from Table F-2. a, = 0.4068 a, = 0.1678 a,, = 0.0900 a16 = 0.0287 a, = 0.2813 a, = 0.1496 a,, = 0.0776' a,, = 0.0172 a, = 0.2415 a, = 0.1331 a13 = 0.0645 a18 = 0.0057 a, = 0.2121 a, = 0.1179 al, = 0.0523 a, = 0.1883 a,, = 0.1036 aU = 0.0404

000935 . 5644

FEMP Background Study May 1994

Table F-4 Example Data Set Number 1 Validated Concentration used result, Midation in stat1 1 Chemical mg/L qualifier Normal Lognormal Barium 0.034 UJ 0.01 7 -4.075 Barium 0.049 U 0.0245 -3.709 Barium 0.05 U 0.025 -3.609 Barium 0.05 U 0.025 -3.689 Barium 0.05 U 0.025 -3.609 Barium 0.05 U 0.025 -3.689 Barium 0.064 U 0.032 -3.442 Barium 0.035 0.035 -3.352 Barium 0.035 0.035 -3.352 Barium 0.039 0.039 -3.244 Barium 0.04 0.04 -3.219 Barium 0.04 0.04 -3.219 Barium 0.04 J 0.04 -3.219 Barium 0.04 J 0.04 -3.219 Barium 0.043 0.043 -3.147 Barium 0.044 0.044 -3.124 Barium 0.044 0.044 -3.124 Barium 0.045 0.045 -3.101 Barium 0.045 J 0.045 -3.101 Barium 0.047 0.047 -3.058 Barium 0.048 0.048 -3.037 Barium 0.049 J 0.049 -3.016 Barium 0.05 0.05 -2.996 Barium 0.051 4 J 0.051 4 -2.968 Barium 0.052 J 0.052 -2.957 Barium 0.054 J 0.054 -2.919 Barium 0.055 0.055 -2.900 Barium 0.057 0.057 -2.865 Barium 0.06 0.06 -2.813 Barium 0.06 0.06 -2.813 Barium 0.061 0.061 -2.797 Barium 0.061 J 0.061 -2.797 Barium 0.062 0.062 -2.781 Barium 0.064 0.064 -2.749 Barium 0.066 0.066 -2.718 Barium 0.073 J 0.073 -2.617 Total 1.634 -1 13.201

000936 F-1 1 FEMP Background Study May 1994 5. Compute W.

1 = - [0.4068 (0.073 - 0.017) + 0.2813 (0.066 - 0.0245) + 0.2415 (0.064 - 0.025) d + 0.2121 (0.062 - 0.025) + 0.1883 (0.061 - 0.025) + 0.1678 (0.061 - 0.025) + 0.1496 (0.060 - 0.032) + 0.1331 (0.06 - 0.035) + 0.1179 (0.057 - 0.035) + 0.1036 (0.055 - 0.039) + 0.0900 (0.054 - 0.04) + 0.0770 (0.052 - 0.04) + 0.0645 (0.0514 - 0.040) + 0.0523 (0.050 - 0.040) + 0.0404 (0.049 - 0.043) + 0.0287 (0.048 - 0.044) + 0.0172 (0.047 - 0.044) + 0.0057 (0.045 - 0.045)12 - (0.080)2 0.0066 = 0.971

6. Fail to reject H, at the 0.05 significance level because W,, = 0.971 is greater than Wcriticalof 0.935, the quantile given in Table F-3, and conclude that the data were drawn from a population with an underlying normal distribution.

The W test was repeated after the transformation of yi = In resulted in W,,. = 0.928. Because the calculated value is less than the critical value (WrritiFl= 0.935), there is sufficient evidence to reject the null hypothesis. The conclusion is that the data set was not drawn from a population having an underlying lognormal distribution.

When the W test fails to reject the null hypotheses for both the normal and lognormal distributions, the W,,, value which exceeds the Wrritidvalue the most is selected as the distribution for the data set. When the .W test rejects the null hypotheses for both the 5644

FEMP Background Study May 1994

normal and lognormal distribution but one is close (W,,, > 0.95 Wcritia,)to the test value, then this distribution (normal or lognormal) is selected. If both the normal and lognormal W,,, values are not close to Wcritia,,then the distribution is considered to be undefined and nonparametric statistical analyses will be used to described the data set.

000938 F-13

'. .

. '. , .. :. FEW Background Study May 1994 a Shapiro-Francia Test for Normality The W test developec by Shapiro and Wilk was used to determine u,ether or not a data set has been drawn from a population which is normally distributed for sample size of 50 or less while the Shapiro-Francia test (Shapiro and Francia, 1972) was used when the sample size was greater than 50.

Like the Shapiro-Wilk test, the Shapiro-Francia test statistic (Wl) can be calculated using the natural logarithm of each data value. This is used to determine whether or not the sample was drawn from an underlying lognormal distribution. The null hypothesis to be tested is:

H,: The population has normal (lognormal when data is transformed) distribution. versus

HA: The population does not have a normal (lognormal when data is transformed) distribution. If H, is rejected, then HA is accepted. To calculate the test statistic one can use the following formula:

where represents the ithordered value of the sample and where m, denotes the approximate expected value of the ithordered normal quantile. The value for m, can be approximately computed as:

F-14 5644

FEW Background Study May 1994 where @-' denotes the inverse of the standard normal distribution with zero mean and unit variance. These values can be computed by hand using a normal probability table (Table F-5) or via simple commands in many statistical computer packages.

1. Order the n data points in assending order (smallest to largest) such that x1

3. Compute the normal quantile (mi).

4. Compute the square of the normal quantile.

5. Compute the multiplication of the normal quantile times the corresponding data value.

6. Compute the arithmetic standard deviation.

n c (3 -32 i=l s= n-1 where:

7. Compute the denominator of Shapiro-Francia Test.

8. Compute the numerator of the Shapiro-Francia Test. 9. Compute W'.

10. Reject H, at the Q significance level is W I is less than the quantile given in Table F-6.

F-15 FEMP Background Study May 1994

Table F-5 Standard Normal Curve for a 2 Distribution

z 0 0.01 0.02 0.03 0.04 0.05 0.06 0.07 0.08 0.09 -3.4 0.0003 0.0003 0.0003 0.0003 0.0003 0.0003 0.0003 0.0003 0.0003 0.0002 -3.3 0.0005 0.0005 0.0005 0.0004 O.OOO4 0.0004 0.0004 0.0004 0.0004 0.0003 -3.2 0.0007 0.0007 0.0006 0.0006 0.0006 O.OOO6 0.0006 0.0005 0.0005 0.0005 -3.1 0.0010 0.0009 0.0009 0.0009 0.0008 0.0008 0.0008 0.0008 0.0007 0.0007 -3.0 0.0013 0.0013 0.0013 0.0012 0.0012 0.001 1 0.001 1 0.001 1 0.0010 0.0010 -2.9 0.0019 0.0018 0.0018 0.0017 0.0016 0.0016 0.0015 0.0015 0.0014 0.0014 -2.8 0.0026 0.0025 0.0024 0.0023 0.0023 0.0022 0.0021 0.0021 0.0020 0.0019 -2.7 0.0035 0.0034 0.0033 0.0032 0.0031 0.0030 0.0029 0.0028 0.0027 0.0026 -2.6 0.0047 0.0045 0.0044 0.0043 0.0041 0.0040 0.0039 0.0038 0.0037 0.0036 -2.5 0.0062 0.0060 0.0059 0.0057 0.0055 0.0054 0.0052 0.0051 0.0049 0.0048 -2.4 0.0082 0.0080 0.0078 0.0075 0.0073 0.0071 0.0069 0.0068 0.0066 0.0064 -2.3 0.0107 0.0104 0.0102 0.0099 0.0096 0.0094 0.0091 0.0089 0.0087 0.0084 -2.2 0.0139 0.0136 0.0132 0.0129 0.0125 0.01 22 0.0119 0.01 16 0.01 13 0.01 10 -2.1 0.0179 0.0174 0.0170 0.0166 0.0162 0.0158 0.0154 0.0150 0.0146 0.0143 -2.0 0.0228 0.0222 0.0217 0.0212 0.0207 0.0202 0.0197 0.0192 0.0188 0.0183 -1.9 0.0287 0.0281 0.0274 0.0268 0.0262 0.0256 0.0250 0.0244 0.0239 0.0233 -1.8 0.0359 0.0351 0.0344 0.0336 0.0329 0.0322 0.0314 0.0307 0.0301 0.0294 -1.7 0.0446 0.0436 0.0427 0.0418 0.0409 0.0401 0.0392 0.0384 0.0375 0.0367 -1.6 0.0548 0.0537 0.0526 0.0516 0.0505 0.0495 0.0485 0.0475 0.0465 0.0455 -1.5 0.0668 0.0655 0.0643 0.0630 0.0618 0.0606 0.0594 0.0582 0.0571 0.0559 -1.4 0.0808 0.0793 0.0778 0.0764 0.0749 0.0735 0.0721 0.0708 0.0694 0.0681 -1.3 0.0968 0.0951 0.0934 0.0918 0.0901 0.0885 0.0869 0.0853 0.0838 0.0823 -1.2 0.1151 0.1 131 0.1112 0.1093 0.1075 0.1056 0.1038 0.1020 0.1003 0.0985 -1.1 0.1357 0.1335 0.1314 0.1292 0.1271 0.1251 0.1230 0.1210 0.1 190 0.1170 -1.0 0.1587 0.1562 0.1539 0.1515 0.1492 0.1469 0.1446 0.1423 0.1401 0.1379 -0.9 0.1841 0.1814 0.1788 0.1 762 0.1736 0.1711 0.1685 0.1660 0.1635 0.1611 -0.8 0.2119 0.2090 0.2061 0.2033 0.2005 0.1977 0.1949 0.1922 0.1894 0.1867 -0.7 0.2420 0.2389 0.2358 0.2327 0.2296 0.2266 0.2236 0.2206 0.2177 0.2148 -0.6 0.2743 0.2709 0.2676 0.2643 0.261 1 0.2578 0.2546 0.251 4 0.2483 0.2451 -0.5 0.3085 0.3050 0.3015 0.2981 0.2946 0.2912 0.2877 0.2843 0.281 0 0.2776 -0.4 0.3446 0.3409 0.3372 0.3336 0.3300 0.3264 0.3228 0.3192 0.31 56 0.3121 -0.3 0.3821 0.3783 0.3745 0.3707 0.3669 0.3632 0.3594 0.3557 0.3520 0.3483 -0.2 0.4207 0.4168 0.4129 0.4090 0.4052 0.4013 0.3974 0.3936 0.3897 0.3859 -0.1 0.4602 0.4562 0.4522 0.4483 0.4443 0.4404 0.4364 0.4325 0.4286 0.4247 -0.0 0.5000 0.4960 0.4920 0.4880 0.4840 0.4801 0.4761 0.4721 0.4681 0.4641 0.0 0.5000 0.5040 0.5080 0.5120 0.5160 0.5199 0.5239 0.5279 0.5319 05359 0.1 0.5398 0.5438 0.5478 0.5517 0.5557 0.5596 0.5636 0.5675 0.5714 0.5753 0.2 0.5793 0.5832 0.5871 0.5910 0.5948 0.5987 0.6026 0.6064 0.6103 0.6141 0.3 0.6179 0.6217 0.6255 0.6293 0.6331 0.6368 0.6406 0.6443 0.6480 0.6517 0.4 0.6554 0.6591 0.6628 0.6664 0.6700 0.6736 0.6772 0.6808 0.6844 0.6879 0.5 0.6915 0.6950 0.6985 0.7019 0.7054 0.7088 0.7123 0.7157 0.7190 0.7224 0.6 0.7257 0.7291 0.7324 0.7357 0.7389 0.7422 0.7454 0.7486 0.7517 0.7549 0.7 0.7580 0.7611 0.7642 0.7673 0.7704 0.7734 0.7764 0.7794 0.7823 0.7852 0.8 0.7881 0.7910 0.7939 0.7967 0.7995 0.8023 0.8051 0.8078 0.8106 0.8133 0.9 0.8159 0.8186 0.8212 0.8238 0.8264 0.8289 0.8315 0.8340 0.8365 0.8389 , p .p;”* < .

CI N/OU5RI /TABF5.XLS/5-94 F-16 000941 FEMP Background Study May 1994

Tablc F-5 (Continued) Stands rd Nom I Curve for a 2 Distribution - - -Z 0 pr 0.02 -0.03 0.04 0.05 0.06 0.07 0.08 0.09 1.o 0.8413 0.8438 0.8461 0.8485 0.8508 0.8531 0.8554 0.8577 0.8599 0.8621 1.1 0.8643 0.8665 0.8686 0.8708 0.8729 0.8749 0.8770 0.8790 0.881 0 0.8830 1.2 0.8849 0.8869 0.8888 0.8907 0.8925 0.8944 0.8962 0.8980 0.0997 0.901 5 1.3 0.9032 0.9049 0.9066 0.9082 0.9099 0.9115 0.91 31 0.91 47 0.9162 0.9177 -1.4 0.91 92 ~ 0.9207 0.9222 -0.9236 0.9251 0.9265 0.9279 0.9292 0.9306 0.931 9 1.5 0.9332 0.9345 0.9357 0.9370 0.9382 0.9394 0.9406 0.9418 0.9429 0.9441 1.6 0.9452 0.9463 0.9474 0.9484 0.9495 0.9505 0.9515 0.9525 0.9535 0.9545 1.7 0.9554 0.9564 0.9573 0.9582 0.9591 0.9599 0.9608 0.9616 0.9625 0.9633 1.8 0.9641 0.9656 0.9664 0.9671 0.9678 0.9686 0.9693 0.9699 0.9706 -1.9 0.971 3 0.9726 -0.9732 0.9738 0.9744 0.9750 0.9756 0.9761 0.9767 2.0 0.9772 0.9778 0.9783 0.9788 0.9793 0.9798 0.9803 0.9808 0.981 2 0.981 7 2.1 0.9821 0.9826 0.9830 0.9834 0.9838 0.9842 0.9846 0.9850 0.9854 0.9857 2.2 0.9861 0.9868 0.9871 0.9875 0.9878 0.9881 0.9884 0.9887 0.9890 2.3 0.9893 0.9896 0.9898 0.9901 0.9904 0.9906 0.9909 0.9911 0.9913 0.991 6 -2.4 0.991 8 0.9920 0.9922 -0.9925 0.9927 0.9929 0.9931 0.9932 0.9934 0.9936 2.5 0.9938 0.9941 0.9943 0.9945 0.9946 0.9948 0.9949 0.9951 0.9952 2.6 0.9953 0.9955 0.9956 0.9957 0.9959 0.9960 0.9961 0.9962 0.9963 0.9964 2.7 0.9965 0.9966 0.9967 0.9968 0.9969 0.9970 0.9971 0.9972 0.9973 0.9974 2.8 0.9974 0.9975 0.9976 0.9977 0.9977 0.9978 0.9979 0.9979 0.9980 0.9981 -2.9 0.9981 0.9982 0.9982 -0.9983 0.9984 0.9984 0.9985 0.9985 0.9986 0.9986 3.0 0.9987 0.9987 0.9987 0.9988 0.9988 0.9989 0.9989 0.9989 0.9990 0.9990 3.1 0.9990 0.9991 0.9991 0.9991 0.9992 0.9992 0.9992 0.9992 0.9993 0.9993 3.2 0.9993 0.9993 0.9994 0.- 0.9994 0.9994 0.9994 0.9995 0.9995 0.9995 3.3 0.9995 0.9995 0.9995 0.9996 0.9996 0.9996 0.9996 0.9996 0.9996 0.9997 -3.4 0.9997 0.9997 0.9997 -0.9997 0.9997 0.9997 0.9997 0.9997 0.9997 0.9998 Source: Devore, J.L., 1982. "Probability & Statistics for Engineering and the Sciences", Table A-3, Brooks/Cole Publishing Company, Monterey, CA.

000942 ._ ClN/OU5Rl/rABFSlX~S/5-94 F-17 FEMP Background Study May1994

TABLE F-6

PERCENTAGE POINTS OF THE WI TEST FOR n > 50

~ ~~ n 0.01 0.05 51 0.935 0.954 53 0.938 0.957 55 0.940 0.958 57 0.944 0.96 1 59 0.945 0.962 61 0.947 0.963 63 0.947 0.964 65 0.948 0.965 67 0.950 0.966 69 0.95 1 0.966 71 0.953 0.967 73 0.956 0.968 75 0.956 0.969 77 0.957 0.969 79 0.957 0.970 81 0.958 0.970 83 0.960 0.97 1 85 0.96 1 0.972 87 0.96 1 0.972 89 0.961 0.972 91 0.962 0.973 93 0.963 0.973 95 0.965 0.974 97 0.965 0.975 99 0.967 0.976

~~~ ~~ Source: Table A-3, U.S. EPA 1992.

CIN/OUSRI/WP/419195/TableF-7/5-94 F-18

.. ‘I . ”* I . -. 000943 FEW Background Study May 1994

To test the null hypothesis that the population has a lognormal distribution, transform the observed data to yi,y2...yn where yi = In xi. Repeat steps 1 through 10 as described above.

Example:

To illustrate the application of the Shapiro-Francia test, the data in Table F-7is used. 1. Order the n data points in ascending order (smallest to largest) as in Column 2.

2. Compute (i/(n+ 1)) where n is the total number of samples (n = 83) which is presented in Column 3.

3. Compute the normal quantile (q)corresponding to Column 3 probabilities. The normal quantiles are presented in Column 4.

4. Compute the square of the normal quantiles which are presented in Column 5.

5. Compute the multiplication of the normal quantiles (Column 4) times the corresponding data point (Column 2). These results are presented in Column 6.

6. Compute the arithmetic standard deviation.

s = 0.019 (See arithmetic standard deviation example for the procedure for calculation of this value; note that the data sets are not the same.)

7. Compute the denominator of the Shapiro-Francia Test.

n (n-1) s2 rnf = 82 (0.019)* (75.611) i=l = 2.238

F-19 000944 FEMP Background Study May 1994 0 8. Compute the numerator of the Shapiro-Francia Test.

9. Compute W*. w' = -0.55 1 2.238

= 0.246

10. Reject H, at the 0.05 significance level because W I = 0.246 is less than Weritifalof 0.985, the quantile given in Table F-6, and concluded that the data were not drawn from a population with an underlying normal distribution.

The Shapiro-Francia Test was repeated after the transformation of yi = In resulted in W I test 7 0.659. Because the calculated value is less than the critical value (Wcritica,= 0.985), it is concluded that the data set was not drawn from a population having an underlying lognormal distribution.

When the W I test fails to reject the null hypotheses for both the normal and lognormal distributions, the W test value which exceeds the W I critifal value the most is selected as the distribution for the data set. When the W test rejects the null hypotheses for both the normal and lognormal distribution but one is close (W I > 0.95 W Icritical) to the test value, then this distribution is (normal or lognormal) selected. If both the normal and lognormal W test are not close to W I critical, then the distribution is considered to be undefined and nonparametric statistical analyses will be used to describe the data set.

F-20 FEMP Background Study May 1994

Table F-7 Example Data Set Number 2 - Normal -Rank -xi (i/(n+ 1)) quantile, mi mi-2 mi*xi 1 0.001 0.01 2 -2.260 5.108 -0.002 2 0.001 0.024 -1 981 3.923 -0.002 3 0.001 0.036 -1.803 3.250 -0.002 4 0.001 0.048 -1.668 2.784 -0.002 5 0.001 0.060 -1.559 2.430 -0.002 6 0.001 0.071 -1.465 2.147 -0.001 7 0.001 0.083 -1-383 1.913 -0.001 8 0.001 0.095 -1.309 1.714 -0.001 9 0.001 0.107 -1.242 1.542 -0.001 10 0.001 0.119 -1.180 1.392 -0.001 11 0.001 0.131 -1.122 1.259 4.001 12 0.001 0.143 -1 .om 1.140 -0.001 13 0.001 0.155 -1.016 1.033 4.001 14 0.001 0.167 -0.967 0.936 -0.001 15 0.001 0.179 -0.921 0.848 -0.001 16 0.001 0.190 -0.876 0.768 -0.001 17 0.001 0.202 -0.833 0.694 -0.001 18 0.001 0.214 -0.792 0.627 -0.001 19 0.001 0.226 -0.751 0.565 -0.001 20 0.001 0.238 -0.71 2 0.508 -0.001 21 0.001 0.250 -0.674 0.455 -0.001 22 0.001 0.262 -0.637 0.406 -0.001 23 0.001 0.274 -0.601 0.362 -0.001 24 0.001 0.286 -0.566 0.320 -0.001 25 0.001 0.298 -0.531 0.282 -0.001 26 0.001 0.31 0 -0.497 0.247 0.000 27 0.001 0.321 -0.464 0.215 0.000 28 0.001 0.333 -0.431 0.186 0.000 29 0.001 0.345 -0.398 0.159 0.000 30 0.001 0.357 -0.366 0.134 0.000 31 0.001 0.369 -0.334 0.1 12 0.000 32 0.001 0.381 -0.303 0.092 0.000 33 0.001 0.393 -0.272 0.074 0.000 34 0.001 0.405 -0.241 0.058 0.000 35 0.001 0.41 7 -0.210 0.044 0.000 36 0.001 0.429 -0.180 0.032 0.000 37 0.001 0.440 -0.150 0.022 0.000 38 0.001 0.452 -0.120 0.014 0.000 39 0.001 0.464 -0.090 0.008 0.000 40 0.001 0.476 -0.060 0.004 0.000 41 0.001 0.488 -0.030 0.001 0.000 42 0.001 0.500 0.000 0.000 0.000 43 0.001 0.512 0.030 0.001 0.000 44 0.001 0.524 0.060 0.004 0.000 45 0.001 0.536 0.090 0.008 0.000 46 0.001 0.548 0.120 0.014 0.000 47 0.001 0.560 0.150 0.022 0.000 48 0.001 0.571 0.180 0.032 0.000 49 0.001 0.583 0.21 0 0.044 0.000 50 0.001 0.595 0.241 0.058 0.000

000946 CI N/OU5R1/419195/rABF6.X~S/5-~ F-21 FEMP Background Study May 1994

Table F-7 (Continued) Example Data Set Number 2 - ____ Normal -Rank -xi (i/(n+ 1)) quantile, mi mi-2 ml*xi 51 0.001 0.607 0.272 0.074 0.000 52 0.001 0.61 9 0.303 0.092 0.000 53 0.001 0.631 0.334 0.112 0.000 54 0.001 0.643 0.366 0.134 0.000 55 0.001 4 0.655 0.398 0.159 0.001 56 0.0015 0.667 0.431 0.186 0.001 57 0.001 5 0.679 0.464 0.215 0.001 58 0.0016 0.690 0.497 0.247 0.001 59 0.002 0.702 0.531 0.282 0.001 60 0.002 0.71 4 0.566 0.320 0.001 61 0.0025 0.726 0.601 0.362 0.002 62 0.0025 0.738 0.637 0.406 0.002 63 0.0025 0.750 0.674 0.455 0.002 64 0.0025 0.762 0.712 0.508 0.002 65 0.0026 0.774 0.751 0.565 0.002 66 0.003 0.786 0.792 0.627 0.002 67 0.004 0.798 0.833 0.694 0.003 68 0.004 0.810 0.876 0.768 0.004 69 0.004 0.821 0.921 0.848 0.004 70 0.0044 0.833 0.967 0.936 0.004 71 0.005 0.845 1.016 1.033 0.005 72 0.005 0.857 1.068 1.140 0.005 73 0.005 0.869 1.122 1.259 0.006 74 0.005 0.881 1.180 1.392 0.006 75 0.006 0.893 1.242 1.542 0.007 76 0.006 0.905 1.309 1.714 0.008 77 0.009 0.91 7 1.383 1.913 0.012 78 0.01 1 0.929 1.465 2.147 0.016 79 0.028 0.940 1.559 2.430 0.044 80 0.029 0.952 1.668 2.784 0.048 81 0.06 0.964 1 .a03 3.250 0.108 82 0.08 0.976 1.981 3.923 0.158 -83 -0.14 0.988 2.260 5.108 0.31 6 -Total 75.61 1 0.742

000947 F-22 i

FEMP Background Study May 1994

Rosner's Test for Many Outliers

To use Rosner's Test (Gilbert 1987, Equations 15.1 to 15.3) it is necessary to specify an upper limit of the number of potential outliers present. This analysis was performed for up to ten outliers. Rosner's Test requires the calculation of a test statistic Ri+lusing the following equation:

where:

Ri+l= test statistic for deciding whether the i + 1 most extreme values in the complete data set are statistical outliers

i = 0 for the first suspected outlier

i = 1 for the second suspected outlier

i = 9 for the tenth suspected outlier

= sample arithmetic mean of the remaining data set after the i most extreme observations have been deleted

F-23 000948 FEW Background Study May 1994

s(') = sample standard deviation of the remaining data set after the i most extreme observations have been deleted and 5 = jthobservation in the data set

n = total number of observations in the data set

A suspected extreme value is determined to be an outlier if the calculated value of Ri+l exceeds the critical value Ai+l for a sample of size n (Table F-8).

In applying Rosner's Test when there is only one suspected outlier, i = 0 and

and x(O) = suspected outlier

2') = sample arithmetic mean of the n observations including the suspected outlier

s(O) = sample arithmetic standard deviation of the n observations including the suspected outlier

F-24 564%

FEMP Background Study May 1994

Table F-6 Approximate Critical Values Lamda (i+ 1) for Rosnets Generalized ESD Many-Outlier Procedure for alpha = 0.05

n 1 2 3 4 5 10 25 2.82 2.80 2.78 2.76 2.73 2.59 26 2.84 2.82 2.80 2.78 2.76 2.62 27 2.86 2.84 2.82 2.80 2.78 2.65 28 2.88 2.86 2.84 2.82 2.80 , 2.68 29 2.89 2.88 2.86 2.84 2.82 2.71 30 2.91 2.89 2.88 2.86 2.84 2.73 31 2.92 2.91 2.89 2.88 2.86 2.76 32 2.94 2.92 2.91 2.89 2.88 2.78 33 2.95 2.94 2.92 2.91 2.89 2.80 34 2.97 2.95 2.94 2.92 2.91 2.82 35 2.98 2.97 2.95 2.94 2.92 2.84 36 2.99 2.98 2.97 2.95 2.94 2.86 37 3.00 2.99 2.98 2.97 2.95 2.88 38 3.01 3.00 2.99 2.98 2.97 2.89 39 3.03 3.01 3.00 2.99 2.98 2.91 40 3.04 3.03 3.01 3.00 2.99 2.92 41 3.05 3.04 3.03 3.01 3.00 2.94 42 3.06 3.05 3.04 3.03 3.01 2.95 43 3.07 3.06 3.05 3.04 3.03 2.97 44 3.08 3.07 3.06 3.05 3.04 2.98 45 3.09 3.08 3.07 3.06 3.05 2.99 46 3.09 3.09 3.08 3.07 3.06 3.00 47 3.10 3.09 3.09 3.08 3.07 3.01 48 3.1 1 3.10 3.09 3.09 3.08 3.03 49 3.12 3.1 1 3.10 3.09 3.09 3.04 50 3.13 3.12 3.1 1 3.10 3.09 3.05 60 3.20 3.19 3.19 3.18 3.17 3.14 70 3.26 3.25 3.25 3.24 3.24 3.21 80 3.31 3.30 3.30 3.29 3.29 3.26 90 3.35 3.34 3.34 3.34 3.33 3.31 100 3.38 3.38 3.38 3.37 3.37 3.35 150 3.52 3.51 3.51 3.51 3.5i 3.50 200 3.61 3.60 3.60 3.60 3.60 3.59 250 3.67 3.67 3.67 3.67 3.67 3.66 300 3.72 3.72 3.72 3.72 3.72 3.71 350 3.77 3.77 3.77 3.77 3.76 3.76 400 3.80 3.80 3.80 3.80 3.80 3.80 450 3.84 3.84 3.84 3.84 3.83 3.83 500 3.86 3.86 3.86 3.86 3.86 3.86 750 3.95 3.95 3.95 3.95 3.95 3.95 1000 4.02 4.02 4.02 4.02 4.02 4.02 2000 4.20 4.20 4.20 4.20 4.20 4.20 3000 4.29 4.29 4.29 4.29 4.29 4.29 4000 4.36 4.36 4.36 4.36 4.36 4.36 5000 4.41 4.41 4.41 4.41 4.41 4.41 Source: Table A-16, Gilbert 1987.

CIN/OU5RI/419195/rABF8.XLS/5-94 F-25

1 *.<,. I. - . FEW Background Study May 1994 When Rosner's Test is applied in a situation where there are two suspectd outliers, i= 1 and

and

x(') = the second suspected outlier

3') = the sample arithmetic mean after the first suspected outlier x(O) has been deleted from the data set

dl) = the sample arithmetic standard deviation after the first suspected outlier x(O) has been deleted from the data set

Example:

Table F-9 presents example data set number 3 which is used for this example. The highest and lowest posted concentrations of 4850 pg/L and 189 pg/L, respectively, are considered to be potential outliers. To begin with, Rosner's Test is applied to determine whether or not the value of 4850 pg/L is a statistical outlier. In this case, i = 0 and

The da I are best described by a lognormal distribution and as such

where:

Y'O) = x(o)

= In (4850)

= 8.487

F-26

. ..J

000959 FEMP BackgroRgt

Table F-9 Example Data Set Number 3

Validated Concentration used result, Validation in stat .its (a) Chemical ug/L qualifier Normal Lognormal Manganese 189 189 5.242 Manganese 301 30 1 5.707 Manganese 351 35 1 5.861 Manganese 370 370 5.914 Manganese 386 386 5.956 Manganese 422 422 6.045 Manganese 437 437 6.080 Manganese 451 451 6.1 11 Manganese 456 456 6.122 Manganese 481 481 6.176 Manganese 488 J 488 6.190 Manganese 521 521 6.256 Manganese 534 534 6.280 Manganese 535 535 6.282 Manganese 543 543 6.297 Manganese 581 581 6.365 Manganese 615 615 6.422 Manganese 619 J 619 6.428 Manganese 747 J 747 6.61 6 Manganese 766 766 6.641 Manganese 785 785 6.666 Manganese 840 840 6.733 Manganese 941 941 6.847 Manganese 1050 1050 6.957 Manganese 1070 J 1070 6.975 Manganese 1090 1090 6.994 Manganese 1150 1150 7.048 Manganese 1460 1460 7.286 Manganese 1500 J 1500 7.31 3 Manganese 4850 4850 8.487 (a) When the validation qualifier contains a 'U", then one-half of the concentration is used in the statistical calculations.

F-27 FEMP Background Study May 1994

To)= sample arithmetic mean of In x using all 30 observations in the data set

= 6.477

s(O) = sample arithmetic standard deviation of In x using all 30 observations in the data set

= 0.6098

Thus 18.487 - 6.477) R, = 0.6098

= 3.296

Because R, = 3.296 is greater than the critical value of Ai+, = 2.91 (Table F-8) for a significance level of 0.05, Rosner's Test indicates that the observed value of 4850 pg/L is a statistical outlier.

Next Rosner's Test is used to determine whether or not there is sufficient evidence to conclude that the minimum value of 189 pg/L is a statistical outlier.

After deleting the maximum value of 4850 pg/L from the data set, calculate

where:

y") = In (189)

= 5.242

yl) = 6.407

s(l) = 0.4889 i 5-64.4

FEMF' Background Study May 1994 Then

15.242 - 6.4071 R, = 0.489

= 2.38

Because R, = 2.38 is less than the critical value of 1, = 2.89 (Table F-8) for a significance level of 0.05, Rosner's Test indicates that there is insufficient evidence to reject the observed value of 189 pg/L. The value is not a statistical outlier. This test is repeated on the high and low ends until there is insufficient evidence to reject the null hypothesis or 10 outliers are identified. This test identifies outliers but professional judgment was used to determine whether the value should be removed from the data set.

F-29 . 000954 EMF' Background Study May 1994 Data Averaging

Data averaging was conducted when two or more samples were collected at the same sampling location on the same day (Le., duplicates, triplicates, etc.). Multiple samples collected on a particular day and location were averaged to obtain one sample per day at the sampling location. Data averaging was conducted to avoid statistical bias which would have resulted from favoring one day's multiple sampling over one sample on a particular day. The data was separated into data groups by the following: media (Le., Glacial Overburden, Great Miami Aquifer, Great Miami River, and Paddys Run), constituent type (Le., radiological, inorganic, and organic constituents), and filter type (Le., filtered and unfiltered). The following steps were repeated for each of the above data groups:

1. Nondetect values were assigned a value of one-half the detection limit. 2. The data were sorted by three key parameters: 1) chemical name, 2) sample location, and 3) sampling date. 3. A record-by-record comparison was performed to identify records for which all three sort parameters were the same. This step identified multiple samples collected on a particular day for a specific constituent.

4. A sample arithmetic mean was calculated on the records with matching key parameters. 5. If all of the values being averaged were nondetects, the average was also a nondetect. If one or more of the values being averaged were detect values, then the resulting average was also considered to be a detect value.

6. A new line was added to the database for the averages of the matching records. 7. The individual records that were combined to create the average line were removed from the statistical data set.

Example:

The example data set is presented in Table F-10. To simplify this example, only one constituent was selected. Seven ammonia samples were collected from 1988 to 1993. The following steps were performed to calculate the averaged values that were used in the statistical analysis:

1. Nondetect values were assigned a value of one-half the detection limit. This occurred for five of the seven values (i.e., c 1 equals 0.05).

CrN/OUS~/wp/4191%/AppENDIxF/s-W F-30

i A ',>L.. , 000955 FEMP Background Study May 1994

Table F-10 Example Data Set Number 4 - Well Sample Lab No. qualifier Constituent w-1 Alkalinitv as CaC03 3 w-1 05/20/93 120072-2 Alkalinitjl as 3 w-1 06/23/93 I12041 6 Alkalinity as CaC03 233 I:, 3 w-1 Alkalinity as CaC03 230 I ID -3 w-1 Aluminum 1.27 I ID 3 w-1 05/20/93 120072-2 Aluminum 3 w-1 06/23/93 120416 Aluminum 2.141-33 I 1 I:, 3 w-1 06/23/93 120414 Aluminum 1.64 I ID -3 w-1 08/29/88 1092 Ammonia 0.1 I J IN 3 w-1 04/03/89 1178 Ammonia 0.1 1 N 3 w-1 05/20/93 120064-2 U Ammonia 0.1 . UJ N 3 w-1 05/20/93 120068-1 U Ammonia 0.1 UJ D 3 w-1 05/20/93 120072-2 U Ammonia 0.1 UJ T 3 w-1 06/23/93 120416 U Ammonia 0.1 U N 3 w-1 06/23/93 120414 U Ammonia 0.1 I U ID -3 w-1 05/20/93 120068-1 U Antimony 0.005 I U ID 3 w-1 05/20/93 120072-2 U Antimony 3 w-1 06/23/93 120416 uw Antimony 0.005 I:, 3 w-1 uw Antimony 0.005 1 UJ ID -3 w-1 U Arsenic 0.005 I U IN 4 w-1 04/03/89 1178 U Arsenic 0.002 U N 3 w-1 05/20/93 120068-1 B Arsenic 0.0039 D 3 w-1 05/20/93 120072-2 B Arsenic 0.0032 T 3 w-1 06/23/93 120416 BW Arsenic 0.0025 J N 3 w-1 U Arsenic 0.002 I U ID -3 w-1 Barium 0.089 I IN 3 w-1 08/29/88 1092 Barium 0.1 N 4 w-1 04/03/89 1178 Barium 0.0493 N 3 w-1 05/20/93 120068-1 B Barium 0.0884 D 3 w-1 05/20/93 120072-2 B Barium 0.0893 T 3 w-1 06/23/93 120416 B Barium 0.0906 N 3 w-1 B Barium 0.0847 1 ID -3 w-1 U Beryllium 0.002 I U ID 3 w-1 05/20/93 120072-2 U Beryllium 3 w-1 06/23/93 I 120416 U Beryllium 0.002O*Oo2 I u I:, 3 w-1 U Beryllium 0.002 I U ID -3 w-1 Cadmium 0.006 I IN 3 w-1 08/29/88 1092 U Cadmium 0.002 U N 4 w-1 04/03/89 1178 Cadmium 0.0098 N 3 w-1 05/20/93 120068-1 U Cadmium 0.005 U D 3 w-1 05/20/93 120072-2 U Cadmium 0.005 U T 3 w-1 06/23/93 120416 U Cadmium 0.005 U N 3 w-1 U Cadmium 0.005 U D -3 w-1 Calcium 77 N 3 w-1 Calcium 70.1 N 4 w-1 Calcium 61.2 N 3 w-1 Calcium 76.5 D 3 w-1 Calcium 77.1 T 3 w-1 Calcium 72.3 N 3 w-1 Calcium 68.4 -. D -3

F-31 000956 FEW Background Study May 1994 2. The data were sorted by three key parameters: 1) chemical name (ammonia), 2) sample location (W-1), and 3) sampling date. There only was one sampling location for W-1 and W-5; however, multiple sampling locations (wells) were sampled for the glacial overburden and Great Miami Aquifer (5 and 24, respectively).

3. A record-by-record comparison identified the following matches: three samples collected on May 20, 1993 and two samples collected on June 23, 1993. Individual records were identified on August 29, 1988 and April 3, 1989.

4. All samples collected on May 20 and June 23, 1993, had nondetect values of ~0.1 (0.05 for statistics); therefore, the averaged value for these two days was a nondetect value of e 0.1.

5. The averaged values are both nondetects, since all of the data used to calculate the average value were nondetects.

6. New lines were added to the database for these two averaged records from the matching key parameters.

7. The five lines that were combined to create the averaged records were removed from the statistical database.

F-32

00095'7 F" Background Study May 1994 Sample Arithmetic Mean - Normal Distribution

The sample arithmetic mean for a normal distribution is given by:

Example:

Using example data set number 1 presented in Table F-4, the sample arithmetic mean is calculated as follows: - X = C0.017 + 0.0245 + .... + 0.066 + 0.073]/36

= 0.045

000958 FEMP Background Study May 1994 Sample Arithmetic Standard Deviation - Normal Distribution The sample arithmetic standard deviation for a population based on a sample is given by:

n n 2 c (3 -32 42 - [E 31 In i=l - i=l i=l n-1 4 n-1

Example:

Using example data set number 1 presented in Table F-4, the sample arithmetic standard deviation for a normal distribution is calculated as follows where n = 36 and:

n xi2 = (0.017)2 + (0.0245)2 + ....+ (0.066)2 + (0.073)2 i=l

= 0.081

[ $ 3[ = [0.017 +0.0245 + .... + 0.066 + 0.07312 i=l

= 2.670

= 1 0.081 - 2.670136 36 - 1

= 0.014 FEMP Background Study May 1994 Estimated Coefficient of Variation - Normal Distribution

The estimated coefficient of variation for a normal distribution is calculated by the following:

cv = sb

where: - x = arithmetic mean s = sample standard deviation CV = estimated coefficient of variation

Example:

Using example data set number 1 presented in Table F-4, the estimated coefficient of variation for a normal distribution is calculated as follows: e-x = 0.045 s = 0.014

CV = 0.014/0.045 = 0.31 FEMP Background Study May 1994 Estimated Mean of a Lognormal Distribution

The estimated mean of a lognormal distribution (Gilbert 1987, Equation 13.7) can be calculated by using the sample mean (y) and the sample standard deviation (s,.) of the log-transformed data. The formula is as follows:

Example:

Using the data in Table F-4, the estimated mean of a lognormal distribution is calculated as follows where n = 36 and:

n WX,) - i=l [ln(0.017) + In(0.0245) +.....+ln(0.066) Y= n 36 - -113.201 36 = -3.144

+= n-1 n-1 FEW Background Study May 1994

n C p(%)p= h1(0.017)~ + ln(O.O245)* +....+ 1n(0.066)2 + ln(0.073)2 i=l

= 360.063

n p(+)]= ln(0.017) + ln(0.0245) +....+ ln(0.066) + ln(0.073) i=l

= -113.201

360.063 - (-113.201)2/36 36-1

= 0.342

(0.342)2 2 1 = 0.046

F-37 FEMP Background Study May 1994 Estimated Standard Deviation of a Lognormal Distribution

The estimated standard deviation of a lognormal distribution (Gilbert 1987, Equation 13.8) can be calculated by using the estimate for the mean of the lognormal distribution (ii)and the sample standard deviation (3)of the log-transformed data. The formula is as follows:

Example:

Data in Table F-4 were used to calculate the estimated standard deviation of a lognormal distribution. The parameters used in this example were presented in the example for estimating the mean of a lognormal distribution where 0 = 0.046 and 3 = 0.342. Therefore, the estimated standard deviation of a lognormal distribution is calculated as follows:

6 = (k0.046)2 [exp (0.342)2 - 11

= 0.0162

963 5644

FEMP Background Study May 1994 Sample Median - Nonparametric Technique

The true median of an underlying distribution (Gilbert 1989, Equations 13.15 and 13.16) is that value above which and below which half of the distribution lies. The median of any distriubtion, no matter what it's shape, can be estimated by the following:

Sample median = x[(~+~)/~~if n is odd

where:

xpl = i* data value in the ordered data set

Example:

Using example data set number 1 presented in Table F-4, the sample median is calculated as follows where n = 36 and:

- '18 + '19 2

= (0.045 + 0.045)/2

= 0.045

If n were 35 rather than 36, then the median would be at XI8 = 0.045.

000964 F-39 FEW Background Study May 1994 Upper One-sided 95% Confidence Limit - Normal Distribution The mean el)for a sample of size n is referred to as a point estimate of the true but unknown population mean (p). If a second sample of size n is drawn from the sample population, the sample mean &) will most likely not be equal toil. In fact if the sampling process is replicated many times, the sample means themselves will have a distribution. Further, the distribution of means of samples of size n will tend toward a normal distribution, if n is sufficiently large.

The 100 (1-a) Upper Confidence Limit (Gilbert 1987, Equation 11.6) of the population mean (p) can also be calculated. When a = 0.05, the upper one-sided 95 percent confidence limit is:

S 95% UCL, = x' + to*ss,-,- J;; where:

toassa-l = value from the "t" distribution in Table F-11.

It should be noted that the 95 percent confidence limit for a second sample of size n drawn from the same population will most likely not be the same as that for the first sample. In theory if a limit estimate is calculated for the means of a very large set of samples of size n, the true population mean will be less than the limit 95 percent of these limits. If the number of degrees of freedom is not listed in Table F-11, then linear interpolation was performed to obtain the t-value.

F-40 000965 FEMP Backgro

Table F-11 Quantiles of the t Distribution (Values oft Such that loop% of the Distribution Is Less Than tp)

Degrees of t at t at t at t at t at t at t at t at freedom 0.60 0.70 0.80 0.90 0.95 0.975 0.990 0.995 1 0.325 0.727 1.376 3.078 6.31 4 12.706 31.821 63.656 2 0.289 0.61 7 1.061 1.886 2.920 4.303 6.965 9.925 3 0.277 0.584 0.978 1.638 2.353 3.182 4.541 5.841 4 0.271 0.569 0.941 1.533 2.132 2.776 3.747 4.604 5 0.267 0.559 0.920 1.476 2.015 2.571 3.365 4.032 6 0.265 0.553 0.906 1.440 1.943 2.447 3.143 3.707 7 0.263 0.549 0.896 1.415 1.895 2.365 2.998 3.499 8 0.262 0.546 0.889 1.397 1.860 2.306 2.896 3.355 9 0.261 0.543 0.883 1.383 1.833 2.262 2.821 3.250 10 0.260 0.542 0.879 1.372 1.812 2.228 2.764 3.169 11 0.260 0.540 0.876 1.363 1.796 2.201 2.71 8 3.106 12 0.259 0.539 0.873 1.356 1.782 2.179 2.681 3.055 13 0.259 0.538 0.870 1.350 1.771 2.160 2.650 3.01 2 14 0.258 0.537 0.868 1.345 1.761 2.145 2.624 2.977 15 0.258 0.536 0.866 1.341 1.753 2.131 2.602 2.947 16 0.258 0.535 0.865 1.337 1.746 2.120 2.583 2.921 17 0.257 0.534 0.863 1.333 1.740 2.1 10 2.567 2.898 18 0.257 0.534 0.862 1.330 1.734 2.101 2.552 2.87% 19 0.257 0.533 0.861 1.328 1.729 2.093 2.539 2.861 20 0.257 0.533 0.860 1.325 1.725 2.086 2.528 2.845 21 0.257 0.532 0.859 1.323 1.721 2.080 2.51 8 2.831 22 0.256 0.532 0.858 1.321 1.71 7 2.074 2.508 2.819 23 0.256 0.532 0.858 1.319 1.714 2.069 2.500 2.807 24 0.256 0.531 0.857 1.318 1.71 1 2.064 2.492 2.797 25 0.256 0.531 0.856 1.316 1.708 2.060 2.485 2.787 26 0.256 0.531 0.856 1.315 1.706 2.056 2.479 2.779 27 0.256 0.531 0.855 1.314 1.703 2.052 2.473 2.771 28 0.256 0.530 0.855 1.313 1.701 2.048 2.467 2.763 29 0.256 0.530 0.854 1.31 1 1.699 2.045 2.462 2.756 30 0.256 0.530 0.854 1.310 1.697 2.042 2.457 2.750 40 0.255 0.529 0.851 1.303 1.684 2.021 2.423 2.704 60 0.254 0.527 0.848 1.296 1.671 2.000 2.390 2.660 120 0.254 0.526 0.845 1.289 1.658 1.980 2.358 2.61 7 Infinite 0.253 0.524 0.842 1.282 1.645 1.960 2.326 2.576

jource: Table A-2, Gilbert 1987.

.,,:, 000966 >” , ., . , -. ‘.I ClN/OU5RI/WP/419195/rABFll .XLS/5-94 F-41 FEW Background Study May 1994 Example:

Using example data set number 1 in Table F-4, the upper one-sided 95 percent confidence limit for a normal distribution was calculated as follows:

n = 36 - X = 0.045

S = 0.014 h.Es = 1.6905

0.014 95% UCL, = 0.045 + (1.6905) -m = 0.049 FEMP Background Study May 1994 Upper One-sided 95% Confidence Limit - hgnormal Distribution The procedure for calculating the upper 95% confidence limit for the lognormal distribution (Gilbert 1987, Equation 13.13) is given by:

2 sy H0.W 95% UCL, = exp + 0.5 sy + - 4x- where:

y =lnx - y = arithmetic mean of y

= sY standard deviation of y n = number of data points

H,,, = value from Table F-12 for sample of size n Example:

The data presented in Table F-4 were used for this example. The mean of the log transformed data (y) equals -3.144 and the standard deviation of the log transformed data equals 0.342. These were previously calculated in the section on calculating the estimated mean of a lognormal distribution. Tables for determining H values are presented individually by number of data points for various significance levels. A summary of these tables for a 0.05 significance level (Ho.95) is presented in Table F-12. When the number of data points was not listed in this Table F-12 then linear interpolation between columns was performed. In addition, if sy did not match the sy in the first column of Table F-12, then linear interpolation between column values was performed. The upper 95% one-sided confidence limit for the lognormal distribution is calculated as follows: where: n = 36 y = -3.144 (see estimated mean of lognormal distribution example) s = 0.342 = 1.807 (linear interpolation between sy values)

F-43 FEMP Background Study

i May 1994

I I I !- I I I

000969 CIN~OlJ5dl/'WP/419195/TABF12.XLS/5-94 F-44 FEMP Backgr &I%!# May 1994 a

6 0c P E

n cQ -P 9 Y.- I r0

F45 FEW Background Study May 1994 .

(0.342) (1.807) 95% UCL, = exp I -3.144 + 0.5(0.342)* + J%=r = exp (-2.981)

= 0.051

Thus the upper one-sided 95% confidence limit for the lognormal distribution is 0.051.

000971 , F-46 .* cIN/OU5~/wp/419195/AppENDIxp/s-94,.. . 5.644.

FEW Background Study May 1994 Upper One-sided 95% Confidence Limit - Nonparametric Technique

The upper 95% one-sided confidence limit for an undefined distribution is based on a noparametric technique (Gilbert 1987, Equation 13.2). It is simply the upper 95% confidence limit on the median of the data set. The following equations are used to calculate the upper one-sided 95% confidence limit for an undefined distribution:

where:

n = number of data points

&.% = upper 95% limit from a standard normal curve for a Z distribution a [Table F-5 at 0.95 (&.,) = 1.6451

U = rank in an ascending order data set that corresponds to the one-sided 95% confidence limit on the median

f(U) = U rounded up to an integer (e.g., 24.2 + 25)

Example:

Using the data presented in Table F-4, the upper one-sided 95% confidence limit on the median is calculated by the following steps.

1. Order the data in ascending order

2. Detennine the number of data points (36)

3. Obtain the &.9s from Table F-5 (1.645)

F-47 FEMP Background Study May 1994 4. Calculate U

U=36+1 + 1.645

= 23.435

5. Round up to an integer (24) 6. Determine the "Uthvalue in the ascending order data set value at the 24th rank = (0.514) The 95% UCL, for this data set is 0.514.

F-48 5644

FEW Background Study May 1994 95* Percentile - Normal Distribution

The 95* Percentile (or Quantile) for a normal distribution are used to determine maximum background concentrations from data sets that are normally distributed. The 95* percentile for a normal distribution (Gilber 1987, Equation 11.1) is calculated based on the following equation:

where: - X = sample arithmetic mean

S = sample arithmetic standard deviation

= upper 95% limit from a standard normal curve for a 2 distribution [Table F-5 at 0.95 (Gags)= 1.6451

Example:

Using example data set number 1 in Table F-4, the 9Sh percentile for a normal distribution was calculated by the following: - X = 0.045

S = 0.014

9SthPercentile, = 0.045 + (1.645) (0.014)

= 0.068

000974 F-49 FEMP Background Study May 1994 95* Percentile - Lognormal Distribution

The 9SthPercentile (or Quantile) for a lognormal distribution (Gilbert 1987, Equation 1 13.24) is calculated based on the following equation.

where: - y = sample arithmetic mean of y

sY = sample arithmetic standard deviation of y

Z,,, Z,,, = upper 95% limit from a standard normal curve for a Z distribution [Table F-5 at 0.95 (Z,,,) = 1.6451 Example:

Using example data set number 1 in Table F-4, the 9Sh Percentile for a lognormal distribution was calculated by the following: - y = -3.144

sY = 0.342 &*, &*, = 1.645

9Sh PercentiZq = exp [-3.144 + (1.645) (0.342)]

= exp (-2.581)

= 0.0757

. .,. ' ' cIN/OUS~/wp/4l9195/AppENDDLF/s-W F-50 FEW Background Study May 1994 9S” Percentile - Nonparametric Technique

The 95* for an undefined distribution is based on a nonparametric technique. It is calculated by the following equations:

Q = 0.95 n

9SthPercentile, = x[f(o)l where:

n = number of data points

Q = rank in an ascending order data set that corresponds to the one-sided 0.95 quantile based on a nonparametric technique

f(Q) = Q rounded up to an integer (e.g., 14.1 + 15) Example:

Using the example data set presented in Table F-4, the 9Sth percentile based on a nonparametric technique is calculated by the following steps:

1. Order the data in ascending order

2. Determine the number of data point (36)

3. Calculate Q

Q = (0.95) (36)

= 34.2

4. Round Q up to an integer (35) 5. Determine the “Qth value in the ascending order data set (value at 3Sth rank = 0.066)

The 9SthPercentile, for this data set is 0.066.

F-5 1 .._:... . .5644 . I.

FEW Background Study May 1994 F-Test

The F-test is conducted to test whether there is no difference between two population variances from a combined data set that is normally distributed. By conducting this test on the natural logarithms of each data value, the F-test was used to determine whether there is no difference between population variances from a combined data set that is lognormally distributed. An alpha of 0.05 was selected for this test. The null hypothesis to be tested is:

H,: The populations have equivalent variances (0: = 0:)

versus

The alternative hypothesis Rejection Region for a Level 0.05 Test

The following is the procedure to determine whether the two groups have variances which are statistically significantly different.

1. Calculate the sample variances of the two groups (si and si).

2. Calculate the test statistic.

3. Determine the two Fcritica,values. These critical values are FOeOu,n1 1, n2 1, which is obtained from Table F-13,while FOaWs,n1 1, n2 is obtained from 1/Fo.a,n2 - 1, "1- 1. 4. Compare Ftatversus the Fcritica,values.

Based on the rejection table listed above, when Fta, is between the two Fcritica,values, there is insufficient information to conclude that the sample variances are from two different populations. Some statistical packages provide p-values (significance levels) based on the F-test statistic (Ftat). When using these statistical packages, a p-value (significance level) of 0.05 was used. When the p-value was greater than or equal to 0.05, then the null hypothesis (H,) was not rejected. When the p-value was less than 0.05, then the null hypothesis was rejected or the alternative hypothesis was accepted.

cIN/OUS~/wp/sl91%/AppENDIX.F/s-94 F-52

1 ~ :'. .: 5644 FEMP Background Study. May 1994

.. . CIN/OU5RI/WP/419195/rABFl3.XLS/5-94 F-53 5644 FEMP Background Study May 1994 Example:

The log transformed data column from Table F-14 was used to conduct the F-test. The procedure for determining whether the two populations have the same sample variances was calculated as follows:

1. Calculate the sample variances.

s2 = -1 c (3 n-1 i=l -a2

s: = 1.415

si = 1.853

2. Calculate the test statistic.

22 Ftcst = Slb2 = 1.41511.853 = 0.764

3. Determine the two Fcritica,values.

F0.Ms,9.14 = 3'21

F0.975,9,14 = 1/F,.~,~4,9 = 113.80 = 0.263 :- 5644 FEMP Background Study May 1994

Table F-14 Example Data Set Number 5

FEk 1 PriW Conc. used No. of Total in stat tics (a) data Uranium, Val idation Log- Log- , points ug/L qualifier Normal normal Normal normal 1 0.1 UJ 0.05 -2.996 0.05 -2.996 2 0.5 J 0.5 -0.693 0.1 UJ 0.05 -2.996 3 0.7 J 0.7 -0.357 0.1 UJ 0.05 -2.996 4 1 1 O.OO0 0.1 UJ 0.05 -2.996 5 1.1 1.1 0.095 0.1 U 0.05 -2.996 6 1.2325 1.2325 0.209 0.55 U* 0.275 -1.291 7 1.791 1.791 0.583 0.55 U* 0.275 -1.291 8 2 J 2 0.693 0.476 J 0.476 -0.742 9 2 2 0.693 0.735 J 0.735 -0.308 10 3.852 3.852 1.349 0.756 J 0.756 -0.280 11 0.851 J 0.851 -0.161 12 1 J 1 0.000 13 1 J 1 0.000 14 1 1 0.000 15 1.5 1.5 0.405 Averaae 1.423 -0.042 0.541 -1.243 (a) When the validation qualifier contains a "u", then one-half of the concentration is used in the statistical calculations.

F-55 FEW Background Study May 1994 a 4. Compare Ftatversus the Fcriticalvalues.

F0.975, 9, 14 ' Ftest and F0.025, 9, 14 > Fta 0.263 c 0.764 and 3.21 > 0.764

Since Fta is between the two Fcriticalvalues, the null hypothesis (H,) is not rejected. If F, were not between the Fcritica,values, the null hypothesis would be rejected. Based on these results, there is insufficient evidence to conclude that the sample variances are from two different populations (lognormally distributed).

a FEMP Background Study May 1994 T-Test

The T-test is conducted to test whether there is no difference between two population means with equal variances from a combined data set that is normally distributed. By conducting this test on the natural logarithms at each data value, the T-test was used to determine whether there is no difference between two population means with equal variances from a combined data set that is lognormally distributed. An alpha of 0.05 was selected for this test. The null hypothesis to be tested is:

Ho: The populations have equal means versus

The alternative hypothesis Rejection Region for a Level 0.05 Test

The following is the procedure to determine whether the two groups have means which are statistically significantly different. 1. Calculate the sample means of the two groups (yland y2).

2. Calculate the sample variances of the two groups (s: and sl>.

3. Calculate the estimated pooled standard deviation.

r 105

4. Calculate the test statistic.

,+-I 1 nl n2

5. Determine the critical values (Tcritica,)from Table F-11. FEMP Background Study May 1994

6. Compare Ttat versus the Tcritidvalues. Based on the rejection table listed above, when the T,, statistic is between the two critical values, there is insufficient information to conclude that the means are from two different populations. Some statistical packages provide p-values (significance levels) based on the Ttatstatistics. When using these statistical packages, a p-value (significance level) of less than 0.05 was used. When the p-value was greater than or equal to 0.05, then the null hypothesis (H,) was not rejected. When the p-value was less than 0.05, then the null hypothesis was rejected or the alternative hypothesis was accepted.

Example:

The log transformed data column from Table F-14 was used to conduct the T-test. The procedure for determining whether the two populations have the same means is as follows:

1. Calculate the sample means. - x1 = -0.042 (from Table F-14) - x2 = -1.243 (from Table F-14) 2. Calculate the sample variances.

sf = 1.415 (from F-test example)

= 1.853 (from F-test example)

3. Calculate the estimated pooled standard deviation.

(10 1)(1.415) + (15 s=[ - - 10 + 15 - 2

= 1.297

F-58 000983 FEMP Background Study May 1994

4. Calculate the test statistic. - (( -0.042) - (-1.243)) Tta - I 1 1.297 J -+- 10 15

= 2.268

5. Determine tk. 3tical values from Table F-11.

6. Compare Tt, versus the critical values.

Ttcst L To.ws,u and Ttest I -To.ws,u 2.268 -> 2.069 and 2.268 -> -2.069 Since T,,, is greater than To.97s,u,the null hypothesis is rejected. Based on these results, there is a statistically significant difference between these two means. Furthermore, 0 these data sets are not from the same lognormal distribution.

0 '- , .. F-59 FEMP Background Study May 1994 The Wilcoxon Rank Sum Test

The Wilcoxon Rank Sum test is a procedure which can be used to determine whether two sample groups have equivalent means. This test assumes that the distributions of the two populations are identical in shape (variance), but the distributions need not be symmetric. The Wilcoxon Rank Sum test was used when comparing two populations (Le., FEMP vs. private wells in the glacial overburden), while the Kruskal-Wallis test was used when comparing three or more populations (Le., Dry Fork, Ross, and Shandon tributaries of the Great Miami Aquifer). In general, the Wilcoxon Rank Sum test should be employed whenever the proportion of nondetects is greater than 15 percent but less than 90 percent. However, in order to provide valid results, the Wilcoxon Rank Sum test should not be used unless both data sets contain at least four samples. The following equations present a step-by-step procedure for conducting the Wilcoxon Rank Sum test.

1. Combine the Group 1 with the Group 2 data and rank the ordered values from 1 to N. Assume there are n Group 1 samples and m Group 2 samples so that N = m + n.

2. Compute the Wilcoxon statistic W: n 1 W= Ei - - n (n + 1) i=l 2

where Eiare the ranks of the Group 1 samples. (Large values of the statistic W give evidence that the groups are not from the same populations.)

3. Compute an approximate Ztat. To find the critical value of W, a normal approxi- mation to its distribution is used. The expected value and standard deviation of W under the null hypothesis (Le., the groups are from the same population) are given by the formulas

1 . E(W) = - mn; 2

An approximate Z,,, for the Wilcoxon Rank Sum test may be calculated by the following equations:

000985 FEMP Background Study May 1994

The factor of 34 in the numerator serves as a continuity correction since the discrete distribution of the statistic W is being approximated by the continuous normal distribution. If n,m > 10 and ties are present, an adjustment to the approximate Z,,, must be made:

g 1 W-E(W) f& - 1) - - j =i %= where SD’ = + 1 - SD’ (W) [TN N(N - 1) and g is the number of tied groups and tj is the number of tied data in the jth group.

4. For a one-tailed 0.05 significance level test for Ho versus the HA (Le., the measurements from population l tend to exceed those from population 2), reject H, and accept HA if 2, 2 ZO4). For a one-tailed a significance level test for H, versus the HA that the measurements from population 2 tend to exceed those from population 1, reject H, and accept HA if 2, -< Z&).

Example:

The data for this example are presented in Table F-15.

1. Combine the FEMP and private Total Uranium data and rank the ordered values from 1 to 25 as shown in Table F-15.

2. Compute the Wilcoxon statistic W n 1 1 W = Ei - -n(n + 1) = 178 - - (10) (11) = 123 i=l 2 2

3. Compute an approximate Z,,,. The expected value and standard deviation of W under the null hypothesis are given by the formulas

E(W) = -1 (15) (10) = 75; 2

F-6 1 FEMP Background Study May 1994

Table F-15 Example Data Set Number 6

Total Uranium Type Overall values used of RZ ranks in calculations (a) well Private 1 0.05 Private 1 2 0.05 Private 2 3 0.05 Private 3 4 0.05 Private 4 5 0.05 Private 5 6 0.05 FEMP N/A 7 0.275 Private 7 8 0.275 Private 8 9 0.476 Private 9 10 0.5 FEMP N/A 11 0.7 FEMP N/A 12 0.735 Private 12 13 0.756 Private 13 14 0.851 Private 14 15 1 Private 15 16 1 Private 16 17 1 Private 17 18 1 FEMP N/A 19 1.1 FEMP N/A 20 1.2325 FEMP N/A 21 1.5 Private 21 22 1.791 FEMP N/A 23 2 FEMP N/A 24 2 FEMP N/A 25 3.852 FEMP N/A Sum 147 (a) These values are from Table F-14.

000987 CIN/OU5R1/419195/TABF15.XLS/5-94 F42

,I 5644

FEMP Background Study May 1994 An approximate Z,,, for the Wilcoxon Rank-Sum test then follows as:

-,v.- 1

4. Compare the approximate && to the upper 95'h percentile of the standard normal distribution = 1.645. Since the approximate Z,,, is greater than 1.645, the null hypothesis may be rejected at the 5 percent significance level, suggesting that there is statistically significant evidence that the populations have different means. The FEMP and private wells for Total Uranium in the Glacial Overburden are not from the same populations and not not be combined.

F-63 FEMP Background Study May 1994 Kruskal-Wallis Test for Comparing Populations

The Kruskal-Wallis test (Gilbert 1987) for comparing populations does not require that data sets be drawn from underlying distributions that are normal or even symmetric, but the K distributions are assumed to be identical in shape. The null hypothesis is

Ho: The populations from which the data sets have been drawn have the same means.

The alternative hypothesis is

HA: At least one population has a mean larger or smaller than at least one other population.

The data can be illustrated as follows:

1 2 3 ... k

x11 X2l '31 ... 'k1

x12 x22 '32 ... xk2 ...

Xln X2" x3n ... 1 2 3 xhk

The total number of data points is m = n1 + n2 + ... + n,; the q need not be equal.

Theses steps must be followed:

1. Rank the m data points from smallest to largest in which the smallest value has rank 1 and the largest value has rank m. If data points are equal (Yies") assign the midrank (e.g., data points 10 and 11 are the same; therefore, 10.5 is used for the rank of both points). If less-than-minimum detectable values occur, treat them as tied values that are less than the smallest detected value that is greater than the minimum detection limit. Suppose the m ranked values are as follows:

F-64 OQ0989 6644

FEMP Background Study May 1994

Assigned Data Value I E? Rank 0.05 1 2 0.05 2 2 0.05 3 2 0.08 4 4 0.09 5 5 0.12 6 6.5 0.12 7 6.5 0.20 8 8 0.21 9 10 0.21 10 10 0.21 11 10

2. Compute Rj the sum of the ranks for each data set.

3. If there are no tied or less-than-minimum detectable values, compute.

where:

m = total number of data values over all data sets Rj = sum of ranks of the jthdata set 9 = number of values in the jth data set k = number of data sets

4. If there are ties or less-than-detectable values, compute:

000990 F-65 FEMP Background Study May 1994 where:

g = number of groups with ties ‘i = the number of tied data in the jth group 5. Reject H, at a level and accept Haif K, (g)is equal to or greater than xl*pl using a probability level of l-a and k-1 degress of freedom taken from Table F-16.

Example:

To illustrate the application of the Kruskal-Wallis & test consider the example data set presented in Table F-16. For these data, m = 30 + 30 + 21 = 81.

1. Rank all 81 observations (Table F-17).

2. The sum of the ranks for each group are as follows (Table F-18).

R, = 1454 R, = 1209.5 R, = 657.5

3. Compute:

(1454)* (1209.5)* + (657.5), + - (82) 81 l2(82) [ 30 30 21 ] ]

= 0.001807 (70,471 + 48,763 + 20,586) - 246

= 252.61 - 246

= 6.61 5644 .: FEMP Background Study May 1994

Table F-16 Quantiles of the Chi-square Distribution with v Degrees o Freedom Degrees of heedom - - -Prob 3il.W a obtain g a va le of c -squre -small -'than te tab1 value - V 0.006 0.010 0.025 0.05 0.100 0.250 0.500 0.750 0.900 0.950 0.975 0.990 0.W ------0.995 ,- 1 0.00 0.00 0.00 0.00 0.02 0.10 0.45 1.32 2.71 3.84 5.02 6.63 7.88 10.83 2 0.01 0.02 0.05 0.10 0.21 0.58 1.39 2.77 4.61 5.99 7.38 9.21 10.60 13.82 3 0.07 0.1 1 0.22 0.35 0.58 1.21 2.37 4.1 1 6.25 7.81 9.35 11.34 12.84 16.27 4 0.21 0.30 0.48 0.71 1.06 1.92 3.36 5.39 7.78 9.49 11.14 13.28 14.86 18.47 5 0.41 0.55 0.83 1.15 1.61 2.67 4.35 6.63 9.24 11.07 12.83 15.09 16.75 20.51 6 0.68 0.87 1.24 1.64 2.20 3.45 5.35 7.84 10.64 12.59 14.45 16.81 18.55 22.a 7 0.99 1.24 1.69 2.17 2.83 4.25 6.35 9.04 12.02 14.07 16.01 18.48 20.28 24.32 8 1.34 1.65 2.1 8 2.73 3.49 5.07 7.34 10.22 13.36 15.51 17.53 20.09 21.95 26.12 9 1.73 2.09 2.70 3.33 4.17 5.90 8.34 11.39 14.68 16.92 19.02 21.67 23.59 27.88 10 2.16 2.56 3.25 3.94 4.87 6.74 9.34 12.55 15.99 18.31 20.48 23.21 25.19 29.59 11 2.60 3.05 3.82 4.57 5.58 7.58 10.34 13.70 17.28 19.68 21.92 24.73 26.76 31.26 12 3.07 3.57 4.40 5.23 6.30 8.44 11.34 14.85 18.55 21.03 23.34 26.22 28.30 32.91 13 3.57 4.1 1 5.01 5.89 7.04 9.30 12.34 15.98 19.81 22.36 24.74 27.69 29.82 34.53 14 4.07 4.66 5.63 6.57 7.79 10.17 13.34 17.12 21.06 23.68 26.12 29.14 31.32 36.12 15 4.60 5.23 6.26 7.26 8.55 11.04 14.34 18.25 22.31 25.00 27.49 30.58 32.80 37.70 16 5.14 5.81 6.91 7.96 9.31 11.91 15.34 19.37 23.54 26.30 28.85 32.00 34.27 39.25 17 5.70 6.41 7.56 8.67 10.09 12.79 16.34 20.49 24.77 27.59 30.19 33.41 35.72 40.79 18 6.26 7.01 8.23 9.39 10.86 13.68 17.34 21.60 25.99 28.87 31.53 34.81 37.16 42.31 19 6.84 7.63 8.91 10.12 11.65 14.56 18.34 22.72 27.20 30.14 32.85 36.19 38.58 43.82 20 7.43 8.26 9.59 10.85 12.44 15.45 19.34 23.83 28.41 31.41 34.1 7 37.57 40.00 45.31 21 8.03 8.90 10.28 11.59 13.24 16.34 20.34 24.93 29.62 32.67 35.48 38.93 41.40 46.80 22 8.64 9.54 10.98 12.34 14.04 17.24 21.34 26.04 30.81 33.92 36.78 40.29 42.80 48.27 23 9.26 10.20 11.69 13.09 14.85 18.14 22.34 27.14 32.01 35.17 38.08 41.64 44.18 49.73 24 9.89 10.86 12.40 13.85 15.66 19.04 23.34 28.24 33.20 36.42 39.36 42.98 25 10.52 11.52 13.12 14.61 16.47 19.94 24.34 29.34 34.38 37.65 40.65 44.31 "5::

26 11.16 12.20 13.84 15.38 17.29 20.84 25.34 30.43 35.56 38.89 41.92 45.64 ~ 48.29 54.05 27 11.81 12.88 14.57 16.15 18.11 21.75 26.34 31.53 36.74 40.1 1 43.19 46.96 49.65 55.48 28 12.46 13.56 15.31 16.93 18.94 22.66 27.34 32.62 37.92 41.34 44.46 48.28 1 50.99 56.89 29 13.12 14.26 16.05 17.71 19.77 23.57 28.34 33.71 39.09 42.56 45.72 49.59 52.34 58.30 30 13.79 14.95 16.79 18.49 20.60 24.48 29.34 34.80 40.26 43.77 46.98 50.89 53.67 59.70 40 20.71 22.16 24.43 26.51 29.05 33.66 39.34 45.62 51.81 55.76 59.34 63.69 73.40 50 27.99 29.71 32.36 34.76 37.69 42.94 49.33 56.33 63.17 67.50 71.42 76.15 186.66 60 35.53 37.48 40.48 43.19 46.46 52.29 59.33 66.98 74.40 79.08 83.30 88.38 99.61

70 43.28 45.44 48.76 51.74 55.33 61.70 69.33 77.58 85.53 90.53 95.02 100.4 ~ 112.3: 80 51.17 53.54 57.15 60.39 64.28 71.14 79.33 88.13 96.58 101.s 106.62 1 12.31 124.W 90 59.20 61.75 65.65 69.13 73.29 80.62 89.33 98.65 107.57 113.15 118.14 124.12 137.2'

100 67.33 70.06 74.22 77.93 82.36 90.13 99.33 109.14 I 18.5C 124.34 129.56 135.81 ~ 149.4! X --2.58 --2.33 --1.96 --1.65 --1.28 --0.67 -0 -0.674 -1.282 -1.645 -1.96 -2.326 2.576 -3.090

For degrees of freedom (v) > 100, chi-squared=v[l-2/9v+X*(2/9v)*0.5]^3 or chi-~quared=0.5*[X+(2v-l)~0.5]~2 if less accuracy is needed, where X is given in the last row of the table. Source: Table A-19, Gilbert 1987.

a ;., 000992

F-67 FEW Background Study May 1994 Table F-17. Ranking of Example Data Set Number 7 (Concentrations in pg/L) Rank Group ChemicalA Rank Group ChemicalA

!I 1 1 189 43 1 521 2 2 251 445 1 534 3 3 288 445) 2 534 45 1 301 46 1 535 45) 2 301 47 1 543 6 2 303 48 3 560 7 3 304 49 2 561 8 3 338 50 2 566 9 3 345 51 1 581 10 2 348 52 2 583 11 1 351 53 1 615 12 3 352 54 1 619 l3 3 356 555 2 621 14 3 358 555) 2 621 15 2 360 57 2 654 16 2 364 58 2 667 175 2 366 59 3 681 175) 2 366 60 2 745 19 1 370 61 1 747 20 2 371 62 3 759 21 3 372 63 1 766 22 3 378 64 1 785 23 1 386 65 2 812 24 2 387 66 2 829 25 2 405 67 1 840 265 2 406 68 3 931 263 3 406 69 1 941 28 3 420 70 2 1020 29 1 422 71 2 1040 30 1 437 72 1 1050 31 1 451 73 1 1070 32 1 456 74 1 1090 33 3 459 75 1 1150 34 3 464 76 2 uoo 35 3 467 77 2 1410 36 3 472 78 1 1460 37 1 481 79 1 1500 38 2 482 80 3 1750 39 3 483 81 1 4850 405 2 486 405) 2 486 42 1 488 Group 1 = Dry Fork Group 2 = Ross \ Group3 = Shandon ) = ties FEMP Background Study May 1994

Table F-18. Ranking of Example Data Set Number 7 by Group (Concentrations in pg/L)

Dry Fork Ross Shandon 1 2 3 45 45 7 11 6 8 19 10 9 23 15 12 29 16 13 30 17.5 14 31 17.5 21 32 20 22 37 24 265 42 25 28 43 265 33 445 38 34 46 405 35 47 405 36 51 445 39 53 49 48 54 50 59 61 52 62 63 555 68 64 555 80 67 57 69 58 72 60 73 65 74 66 75 70 78 71 79 76 81 n SUm 1454 12095 6575

OQ0994 cEN/OUSRI/wp/4 I91%/AF'PENDIXF/SW F-69 -/ ./ EMF' Background Study May 1994 4. Because there are ties, calculate a modified c:

--I 1- 1 E 'j (f - 1) M (m2 - 1) j=1

where:

g = 6 = number of groups with ties tl = 2 = number of data points in the first tlzd group (Rank -3) t2 = 2 = number of data points in the second tied group (Rank 17.5) t3 = 2 = number of data points in the third tied group (Rank 26.5) t4 = 2 = number of data points in the fourth tied group (Rank 46.5) tS = 2 = number of data points in the fifth tied group (Rank 44.5) t6 = 2 = number of data points in the sixth tied group (Rank 55.5) , 6.61 1 1- [ 2(3) + 2(3) + 2(3) + 2(3) + 2(3) + 2~1 81 [(81)2 - 1) - 6.61 0.99993

= 6.61

5. Because = 6.61 is greater than = 5.99, the null hypothesis H,, is rejected at the a = 0.05 level. The conclusion is that at least one population being compared has a mean different from the other populations.

F-70