Computer-Assisted Data Treatment in Analytical Chemometrics III. Data Transformation
aM. MELOUN and bJ. MILITKÝ
department of Analytical Chemistry, Faculty of Chemical Technology, University Pardubice, CZ-532 10 Pardubice ъDepartment of Textile Materials, Technical University, CZ-461 17 Liberec
Received 30 April 1993
In trace analysis an exploratory data analysis (EDA) often finds that the sample distribution is systematically skewed or does not prove a sample homogeneity. Under such circumstances the original data should often be transformed. The power simple transformation and the Box—Cox transformation improves a sample symmetry and also makes stabilization of variance. The Hines— Hines selection graph and the plot of logarithm of the maximum likelihood function enables to find an optimum transformation parameter. Procedure of data transformation in the univariate data analysis is illustrated on quantitative determination of copper traces in kaolin raw.
When exploratory data analysis shows that the i) Transformation for variance stabilization implies sample distribution strongly differs from the normal ascertaining the transformation у = g(x) in which the one, we are faced with the problem of how to analyze variance cf(y) is constant. If the variance of the origi the data. Raw data may require re-expression to nal variable x is a function of the type cf(x) = ^(x), produce an informative display, effective summary, the variance cr(y) may be expressed by or a straightforward analysis [1—10]. We may need to change not only the units in which the data are 2 fdg(x)f stated, but also the basic scale of the measurement. <У2(У)= -^ /i(x)=C (1) To change the shape of a data distribution, we must I dx J do more than change the origin and/or unit of mea surement. Changes of origin and scale mean linear where С is a constant. The chosen transformation transformations, and they leave shape alone. Non g(x) is then the solution of the differential equation linear transformations such as the logarithm and square root are necessary to change shape. This paper brings a description of the power trans formation and the Box—Cox transformation and a re- expression of statistics for transformed data. The In some instrumental methods of analytical and procedure of the power transformation and the Box- physical chemistry, the relative standard deviation Cox transformation is illustrated on a practical ex <5(x) of the measured variable is constant. This ample of the quantitative determination of copper means that the variance
164 Chem. Papers 48 (3) 164-169 (1994) ANALYTICAL CHEMOMETRICS. Ill
x values by (x - x ) values which are always posi x for parameter Я > О 0 tive. Here x0 is the threshold value x0 < x(1). У = flf(x) = In x for parameter Я Ф О (3) An excellent diagnostic tool enabling estimation of
я parameter Я is represented by the Hines—Hines -х~ for parameter Я < О selection graph [8]. It is based on the equation which does not retain the scale, is not always con A -A ( У Л X tinuous and is suitable only for positive x. Optimal X 0.5 PJ + = 2 (S) estimates of parameter Я are sought by minimizing X V 0.5 J X-\_ p the absolute values of particular characteristics of asymmetry. In addition to the classical estimate of valid for distribution symmetrical around a median. a skewness ^(y), the robust estimate g (y) is used 1iR For the cumulative probability P, = 2~* , the letter values F, £, / = 2, 3 are usually chosen. - ,x _ (У0.75 ~Уо.5о) ~ (У0.50 ~ У0.25) м\ To compare empirical dependence of experimen tal points with the ideal one, ideal curves for vari (У0.75-У0.25) ous values of parameter Я are drawn in a selection The robust estimate of asymmetry g (y) may be also graph. These curves Я represent a solution of the P я я expressed with the use of a relative distance between equation у + х~ = 2 in the range 0 < x < 1 and 0
Chem. Papers 48 (3) 164-169 (1994) 165 M. MELOUN, J. MILITKY
After an appropriate transformation of the original On the basis of the (known) actual transformation data {x} has been found, so that the transformed data у = g(x) and the estimates y, s2(y) it is easy to cal give approximately normal symmetrical distribution 2 culate re-expressed estimates xR and s (xR): with constant variance, the statistical measures of 1. For a logarithmic transformation (when Я = 0) location and spread for the transformed data {y} are and g(x) = In x the re-expressed mean and variance calculated. These include the sample mean y, the are calculated by eqns (18) and (19) sample variance s2(y), and the confidence interval 1/2 2 of the mean у ± ^ _ Q/2 (n - 1)s(y)/(n) . These esti x = exp [y + 0.5s (/)] (18) mates must then be recalculated for original data {x}. R and Two different approaches to re-expression of the sta 2 2 2 tistics for transformed data can be simply used: s (xR) = x s (y) (19) 1. Rough re-expressions represent a single reverse transformation x = gr1(y)- This re-expression for a 2. For Я Ф 0 and the Box—Cox transformation (7) R the re-expressed mean x will be represented by one simple power transformation leads to the general R mean of the two roots of the quadratic equation 5X *R,1.2 =[0-5(1 + 1У)± / = 1 1Д (12) ± 0.5^ + 2Цу+ s2(y)) +Г (y2-2s2 (y)) (20)
1 which is closest to the median x0.5 = <Г (У o.s)- If xR where for Я = 0, In x is used instead of Xя and ex is known the corresponding variance may be calcu 1/A lated from instead of x . The re-expressed mean xR = x_-, stands for the harmonic mean, x = x for the geo -(-2A + 2) 2, R 0 s'W = x£"«r{y) (21) metric mean, xR = x, for the arithmetic mean, and xR = x2 for the quadratic mean. 2. The more correct re-expressions are based on the Taylor series expansion of the function у = g(x) COMPUTATION in a neighbourhood of the value y. The re-expressed mean x is then given Procedure POWER TRANSFORM in package R ADSTAT [11] searches parameters of simple power transformation and parameters of normalized Box— 4-2 1 d^(x) dg(x) Cox transformation of data. It enables the explora -1 2 s (y) (13) tory data analysis of transformed data. For the trans 2 dx" dx formation (3) different measures of symmetry (4) and (5) are calculated and the sample curtosis in the For variance it is then valid range -3 < Я < 3 with a step 0.1 and the optimal values of these measures are printed. The selection rdg(x)A 2 graph is drawn as well as the points of optimal val s (*R)~ s (y) (14) dx ues of Я. From this graph the value of Я can be es timated. Using transformed data the mean y, the 2 where individual derivatives are calculated at the variance s (y), the skewness g^y), and the curtosis g (y) are calculated. These computations can be point x = xR. The 100(1 - a) % confidence interval 2 of the re-expressed mean for the original data may repeated for various values of Я. For the transfor be defined as mation (6) the estimate Я maximizing In ЦЯ) defined by eqn (9) is calculated. Different measures of sym X -/ <^ the re-expressed estimates of original variables xR 1 cŕgríx) dg(x) (73), s2(x ) (14), and the 95 % confidence interval G = - s2(y) (17) R 2 dx'' dx of the re-expressed variable ц are calculated. 166 Chem. Papers 48 (3) 164-169 (1994) ANALYTICAL CHEMOMETRICS. Ill RESULTS i i i i i г Study Case 1. Determination of copper trace in kaolin In a standard sample of kaolin the content of cop per trace was determined in ppm and the values were arranged in increasing order. The type of a sample distribution and measures of location and scale were examined. Data: the copper content w/ppm in increasing or der gives a set: 4, 5, 7, 7, 7, 8, 8.3, 8.4, 9.4, 9.5, 10, 10.5, 12, 12.8, 13,22,23. J I I L 2.00 6.00 10.00 14.00 18.00 22.00 Solution: Applying an analysis of basic assump X tions about data the following conclusions were met: Fig. 2. The kernel estimation of the probability density function a) Combined sample skewness and curtosis test of original sample: 1. robust, 2. classical. leads to statistic Сл = 7.908 > /(0.95, 2) = 5.992 and therefore a normality of data distribution was rejected. b) Interval of both Hoaglin's outer bounds [- 3.191; 22.191] does not contain one observation and there fore this point x(17) may be denoted as an outlier. The measures of location, scale and distribution shape for data without 1 outlier are x = 9.619, s(x) = 4.170, g^x) = 1.610, and g2{x) = 6.340. c) Test of sample elements independence leads to statistic r17 = 1.036, ro.975 (18) = 2.101 and there fore an independence is accepted. Examining the first part of the EDA diagnostics following sample properties were found: the jittered 0.00 2.00 dot diagrams and the box-and-whisker plots (Fig. 1) -2.00 indicate two outliers which can be accepted if the G- normal distribution is skewed. Fig. 3. The quantile-quantile (rankit) plot of original sample. The nonparametric kernel estimation of probabil ity density function (Fig. 2) indicate that the distri The second part of EDA concerns the search for bution is skewed towards higher values. The a suitable symmetric transformation of the data. The quantile-quantile (rankit) plot (Fig. 3) with convex selection graph (Fig. 4) shows that the optimal power increasing shape confirms that the distribution is reaches a value above - 0.5 in the range near zero skewed to higher values. which corresponds to a logarithmic transformation. 1.UU I I I I У • • • m шт *m • • 1.00 I I I 1 1 0.80 ... ll.r • 0.80 .. •'•.".» • " '.'.'.'.'.'•'•'•'• 0.60 0.60 ^XV'///// 0.40 I Ч 1 S 0.40 / / / / / • 0.20 • H[X ]h 0.20 / / • i 0.00 1 i 1 0.00 0.00 10.00 20.00 x "1 1 ' 1 Г ' i 0.00 0.40 0.80 axis 1 Fig. 1. The jittered diagrams and the box-and-whisker plots of original sample. Fig. 4. The Hines—Hines selection graph. Chem. Papers 48 (3) 164-169 (1994) 167 M. MELOUN, J. MILITKY - 15.00 I I I I I I -20.00 - -25.00 PS* - 30.00 / \ - 35.00 / \ - 40.00 / \ - 45.00 / \ - 50.00 - \ -55.00 0.00 - "i I I III" -6.00 -2.00 2.00 Л 6.00 •2.00 -1.00 0.00 1.00 2.00 Fig. 5. The plot of logarithm of likelihood function. Q-normal From the plot of the logarithm of the likelihood func tion (Fig. 5) for the Box—Cox transformation the maximum of the curve is at Я = - 0.2. The corre sponding 95 % confidence interval does not contain the value Я = 1, so this transformation is statistically significant. The rankit plot (Fig. 6a—c) shows that there is a significant improvement in the distribution symmetry for transformation Я = - 0.27. The measures of location, spread and shape for the original data have values of mean x = 10.406, standard deviation s(x) = 5.180, skewness g^(x) = 1.399, and curtosis g^(x) = 4.272. After a logarith -2.00 -1.00 0.00 1.00 2.00 mic transformation (A = 0) the values are 2.243, OL- normal 0.203, 0.304, and 3.070, and after a power transfor 2.40 mation (A = - 0.27) they are 0.5536, 0.065, 0.048, and 3.071 while the Box—Cox transformation (A = 2.20 - 0.27) leads to values 1.674, 0.246, - 0.048, and 2.00 3.071. By the rough re-expression (12) xR = exp (x*) = 1.80 9.337. The corresponding confidence limits are /L = «» 1.60 7.742 and lö = 11.878 (eqns (16a, 16b)). Quantile ro.97 (17-1) = 2.12. 5 1.40 By the more correct re-expression (13) there is xR = 9.187 with /L = 8.272 and lu = 13.147 (eqn (20)). 1.20 In comparison of the sample distribution with a ~ / theoretical exponential one, the correlation coefficient 0.00 1.00 2.00 rxy of the 0-0 plot is found to be 0.967, while for the -2.00 -1.00 log-normal one rxy is 0.961. G-normal The assumption of the log-normal distribution is Fig. 6. The quantile-quantile plot indication of improvement of acceptable. Because of the small sample size it is a distribution symmetry of a) original data when b) the difficult to be certain whether there are outliers in power transformation, and c) the Box—Cox transforma the sample, or if the sample distribution is of skewed tion are applied. log-normal or of skewed exponential nature. sures of transformed data are re-transformed to get these unbiased and rigorous measures for original CONCLUSION data. Often, the chemical data are less ideal and do not REFERENCES fulfill all basic assumptions. Original data are then transformed to improve a symmetry of data distri 1. Tukey, J. W., Exploratory Data Analysis. Addison Wesley, bution and a variance stabilization. Statistical mea- Reading, Massachusetts, 1977. 168 Chem. Papers 48 (3) 164-169 (1994) 2. Chambers, J., Cleveland, W., Kleiner, W., and Tukey, P., New York, 1983. Graphical Methods for Data Analysis. Duxbury Press, Bos 7. Kafander, K. and Spiegelman, C. H., Comput. Stat. Data ton, 1983. Anal. 4, 167 (1986). 3. Hoaglin, D.C., Mosteier, F., and Tukey, J. W., Exploring 8. Hines, W. G. S. and Hines, R. J. H., Am. Statist. 41, 21 Data Tables, Trends and Shapes. Wiley, New York, 1985. (1987). 4. Scott, D. W. and Sheater, S. J., Commun. Statist. 14, 1353 9. Hoaglin, D. C, J. Am. Statist. Assoc. 87, 991 (1986). (1985). 10. Stoodley, K., Applied and Computational Statistics. Ellis 5. Lejenne, M., Dodge, Y., and Koelin, E., Proceedings of the Horwood, Chichester, 1984. Conference COMSTAT'82 Toulouse. P. 173 (Vol. ///). 11. Statistical package ADSTAT 2.0. TriloByte, Pardubice, 1992. 6. Hoaglin, D. C, Mosteier, F., and Tukey, J. W. (Editors), Understanding Robust and Exploratory Data Analysis. Wiley, Translated by M. Meloun The Testing of Carbon Sorbent for Preconcentration of Volatile Organic Trace Compounds aS. ŠKRABÁKOVÁ, aE. MATISOVÁ, aM. ONDEROVÁ, bl. NOVÁK, and bD. BEREK ^Department of Analytical Chemistry, Faculty of Chemical Technology, Slovak Technical University, SK-812 37 Bratislava ^Polymer Institute, Slovak Academy of Sciences, SK-842 38 Bratislava Received 30 April 1993 Carbon sorbent Carb I (prepared by controlled pyrolysis of saccharose) was tested for preconcentration of volatile organic compounds from the gas phase. The model mixture of hydro carbons (n-alkanes and aromatics) and mixture of aromatics with low-boiling polar solvents was used. For desorption of compounds several solvents were utilized, carbon disulfide was found to be the best. Adsorption—desorption process was studied in the concentration range of compo nents in nitrogen 0.03—15 |ig dm"3. Chromatographic measurements were performed on gas Chromatograph with on-column and splitless injection, fused silica capillary columns with chemi cally bonded stationary phases under temperature programmed conditions and flame ionization detector. The recovery of n-alkanes and aromatics was found to be around 90 %, the recovery of low-boiling solvents, particularly of polar character was low. The main part of toxicological analyses and analy tion of components of interest, isolation of determined ses that have been required with regard to the con analytes from the matrix and removal of potential inter trol of the environment is concentrating upon com ferences. For this reason there have been utilized each pounds with very low concentration. In many cases time more special qualities of sorbent materials [2—4]. the concentrations of contaminating components are In the selection of a proper sorbent it is neces so low, e.g. in air, that the common detectors used, sary to take into consideration general characteris e.g. in gas chromatography, do not detect them. Or tics as functional groups at the surface, chemical ganic contaminating components in environmental and thermal stability, as well as inertness and cata samples generally occur in ng kg"1 to |ig kg"1 as a lytic properties, mechanical resistance, pores diam part of a complex matrix [1]. Besides that, samples eter and volume, specific surface area, size and are usually not compatible with chromatographic shape of particles. system, therefore analysis with direct sample injec Affinity of sorbents towards various organic com tion is not possible. pounds depends on the type of functional groups It is therefore necessary to perform sample pre- bound on the surface of a sorbent and on their ori treatment before an analysis. It is mainly preconcentra- entation on the surface. Chem. Papers 48 (3) 169-174 (1994) 169