Homepage of Chemometrics

© 1996-2002

http://www.acc.umu.se/~tnkjtg/Chemometrics/Editorial

OUTLINE 1.) Introduction Introduction to Statistical Experimental Design 2.) Stages in the DoE - What is it? Why and Where is it Useful? process 3.) Analysing the resulting experimental Johan Trygg & Svante Wold 4.) Applications of DoE University of Queensland, Australia & Umeå University, Sweden 5.) References 6.) Appendix 7.) DoE Exercises

Definition: Statistical experimental design, a.k.a. design of (DoE) is the methodology of how to conduct and plan experiments in order to extract the maximum

amount of information in the fewest number of runs.

1. Introduction List of previous 1.1 The traditional way - the COST ap- …We have a large reservoir of engineers (and HoC Editorials are proach [COST: Change One Separate scientists) with a vast background of engineering given on last page. factor at a Time] know- how. They need to learn statistical methods that can tap into the knowledge. used as a Most experimentation today is done by changing catalyst to engineering creation will, I believe, always levels of one factor (variable) at a time in an result in the fastest and most economical pro- unsystematic way in order to try and find the gress… George Box, 1992 optimum conditions of a complex system. Is this a good (efficient, rational, economic) strategy ?

No!! How shall I find the optimum? This is a com- As shown by Fisher around 1925, changing one mon question everywhere in business. In re- separate factor at a time (COST) does not give search and development, often half of the re- any information about the position of the opti- sources are spent on solving optimization prob- mum in the common case where there are in- lems. With the rapidly rising costs of making teractions between factors. Then the COST experiments, it is essential that the optimization approach gets stuck, usually far from the real is done with as few experiments as possible. optimum. However, the experimenter perceives This is one important reason why statistical that the optimum has been reached because experimental design is needed. changing one factor at a time does not lead to

any further improvement. The COST approach DoE originated in the 1920's by a British scien- is said to be pseudo-convergent. tist, Sir R. A. Fisher, as a method to maximize

the knowledge gained from experimental data What are the problems associated with and it has evolved over the last 70 years. Most the COST approach: experimentation involves several factors and are conducted in order to optimize processes 1. Does not lead to real optimum and or investigate and understand the relation- 2. Inefficient, unnecessarily many runs ships between the factors and the characteris- 3. Provides no information about what happens tics of the process (reponses) of interest. when factors are varied simultaneously (ignores interactions) 4. Provides less information about the variability http://www.acc.umu.se/~tnkjtg/Chemometrics/Editorial Page 2

Typical examples when design of experi- ments (DoE) is useful involve the devel- opment of new products and processes, e.g. optimizing the quality and perform- ance of an existing product or optimizing existing manufacturing processes of chemicals, polymers, materials, drugs and pharmaceuticals, foods and bever- ages, cosmetics, paints and so on.

2. Stages in the DoE proc- ess 2.1 Familiarization, fiddle around Fiddling around a little. Problem formula- tion is extremely important during the whole process. Formulate question(s) stating the objectives and goals of the investigation. What do I want?

• What is the purpose? • What are the objectives? • Identify factors, factor ranges and types of factors (quantitative or qualitative) • What is possible, experimentally, financially, environmentally? of the response of methods of statistical experimental If these objectives can not be put into 5. Isolated, unconnected experiments design. These methods have been fur- words, there is no reason to continue 6. Slow growth of knowledge, no map- ther refined by Yule, Box, Stu and Bill because it shows that the investigators ping of experimental space. Hunter, Scheffe, Cox, Taguchi, and oth- don't know what to do. ers, so that today they comprise a tool Any measurement and is box for virtually any optimization prob- 2.2 Screening (many factors) influenced by noise. Under stable condi- lem. The basic idea is to devise a small Finding out a little about many factors. tions, any process varies around its set of experiments, in which all pertinent Which factors are the dominating ones. +/- 3 std dev. Two COST experi- factors are varied systematically. This set To assure that uncontrolled factors ments may give two different results, but usually does not include more than ten (humidity, etc..) do not bias the results, with no estimate (poor) of noise level. to twenty experiments. The subsequent perform the runs in random order BUT by making a set of well planned analysis of the resulting experimental Screening experiments give information experiments with correct analysis (DoE), data will identify the optimal conditions, about we can separate "real effects" from noise the factors that most influence the re- • What are the important factors and draw correct conclusions and act sults and those that do not, the presence • If we are in the correct region, of interactions and synergisms, and so correctly. This decreased variabil- (ranges) on. The most important aspect of statis- ity and quality improvement. So to inves- • If there is curvature and if it tigate systems involving several factors in tical experimental designs is that they masks the effects presence of variability or noise one provide a strict mathematical framework • What to do next needs a better strategy than that based for changing all pertinent factors simulta- on changing one separate factor at a neously, and achieve this in a small num- time and that strategy is given by experi- ber of experimental runs. Most of us can Pareto's principle states that 20 % of the mental design (DoE). only grasp the effect of one factor at a data (factors) account for 80 % of the time in our minds, and that leads to the information. Screening designs provide 1.2 What to do instead - Design inefficient COST approach. We need the simple models with information about dominating variables, and information of Experiments (DoE) mathematics (and the computer) to keep • track of the factors and their combina- about ranges. In addition, they provide Which factors have a real influ- tions. few experiments / factor which means ence on the response? • All factors are varied together that relevant information is gained in • What are the best settings of the over a set of experimental runs only a few experiments. factors to achieve optimal condi- • Noise is decreased by means of Linear models and models tions for best performance on a are sufficient, since we are only inter- averaging system? ested in the effects. We merely ask, if a • The functional space is efficiently • What are the predicted values of factor does influence the response, not mapped, interactions and syner- the responses for given settings how. gisms are seen of the factors in a model?

In 1925 Fisher started the development http://www.acc.umu.se/~tnkjtg/Chemometrics/Editorial Page 3

degree interaction effects. This loss of information is the prize we need to pay

2.2.1 Screening designs: 5. Noise can be estimated from -Replicated center points Different types of screening designs -Other for the reduction of the number of ex- exist, which one to choose depends on -Residuals (higher level interac- periments. the problem. The most common one is tions) • Fraction (k-p) selected by choice the fractional factorials design. -Previous knowledge of generator • Factorial designs form the basis for all Degree of of coeffi- classical experimental designs, both cients can be derived from the screening and RSM. For screening, we defining relation The shortest will concentrate on two-level designs. word in the defining relation is They are sufficient to estimate linear, the resolution of that fractional and interaction models, and they require factorial design. a very low number of experimental runs. There are some clear obvious advan- 2.2.4 Placket-Burman tages of using a factorial design. See http://www.itl.nist.gov/div898/ • Averages are more stable than handbook/pri/section3/pri335.htm for single observations. more info.

• The more data one averages, the Figure. Example of a full factorial more reliable is the result. design in three variables, 2.2.5 Special designs 23= 8 experiments In addition, special designs are needed in The "equal opportunity" strategy constrained regions or mixture problems. Mixture problems are common in the (screening) chemical, food and beverage, cosmetics, 1. The factors are selected that a Notation priori are believed to be the and drug industries. One reason is be- The low level of a factor will be denoted cause the factors add up to 100 percent. most influential on the response [-1] = [-] 2. A is chosen for each factor This introduces a constraint on the de- The high level of a factor will be denoted sign and must be handled with special 3. A (full factorial) or fractional [+1] =[+] factorial design (plus 3 ctr. tools and models. D-optimal designs or The center level of a factor will be de- other special designs are used when points) is selected. This provide noted [ 0] = [0] orthogonal, balanced estimates there are constraints put on the factors for economical / synthetical / of the effects (and possibly inter- 2.2.3 Fractional factorial design actions) with equal . environmental or other reasons. The Investigating more than 5 factors with 4. The experimental runs are per- simpler factorial designs can not be used the full factorial design becomes time formed in random order. This in such cases. consuming, 25=32, 26=64,27=128, etc. results in a of the factors • D-optimal (and other "optimal" Instead, performing a fractional factorial (and possibly interactions) with designs) design reduces that number quickly significance estimates. - See http://www.itl.nist.gov/ without the loss of too much informa- Let us first describe the full factorial div898/handbook/pri/section5/ tion regarding the estimation of factors design, and later explain the fractional pri521.htm for more info. involved. Fractional factorial design takes factorial design. • Mixture designs advantage of the fact that 3-way and higher interactions are seldom signifi- - See http://www.itl.nist.gov/ 2.2.2 Full factorial design cant. The downside, of course, for not div898/handbook/pri/section5/ The full factorial design is a set of ex- performing all experiments is that con- pri54.htm for more info perimental runs where every level of a founding patterns are present. In other factor is investigated at both levels of all words, the estimated effects are not 2.3 Finding optimal region of op- the other factors. It is a balanced "pure" but instead mixed with higher erability (orthogonal) design. A balanced design Simplex designs (see http:// allows the estimation of a factor effect www.multisimplex.com/algorithm.htm independently of all the other effects for more info) and steepest ascent 1. N=2k (+3 centerpoints) number approaches (see http://www.itl.nist.gov/ of runs for k factors div898/handbook/pri/section5/ 2. Effects (main effects, interac- pri5311.htm for more info.) are used to tions) are calculated using least achieve optimal conditions in problems squares ( or Yates algorithm) where experiments can only be done 3. Estimate of noise is used for one at a time in a sequence or after the for the ef- screening stage where one is often inter- fects, or hypothesis testing ested into moving the experimental 4. When no estimate of noise is region to its optimum. Figure. Example of a fractional available, normal probability plot factorial design in three variables, of the effects is used. 2.3 Response surface modeling 23-1= 4 experiments

http://www.acc.umu.se/~tnkjtg/Chemometrics/Editorial Page 4 and optimization (few factors) Usually a fractional factorial (see section tions to Latent Structures (PLS) After screening, the goal of the investiga- 2.2.3 in this editorial) or Plackett- PLS is one of the most common meth- tion is usually to create a valid map of Burman design is used. ods for analyzing multivariate data where the experimental domain (local space) a quantitative relationship between a given by the significant factors and their 3. Analysing the resulting descriptor matrix X and a response ranges. This is done with a quadratic matrix Y is sought. The PLS model can polynomial model. The higher order experimental data be expressed by: models has an increased complexity and After the planning stage, when the set of therefore also requires more experi- experiments are laid out according to a Model of X: X = TPT+E ments / factor than screening designs. statistical design, the planned experi- Model of Y: Y = TCt +F Different types of RSM designs ments are made, either in parallel, or • Three level factorial designs one after another. Each experiment gives PLS contains MLR as a special case, then - See http://www.itl.nist.gov/ results, i.e. values of the response vari- the PLS regression coefficients and the div898/handbook/pri/section3/ ables. Thereafter, these data are ana- MLR coefficients are identical. pri339.htm for more info. lysed by means of multiple regression, or • Central composite designs generalisations thereof such as the PLS 3.3 Orthogonal projections to (CCD) and O-PLS methods (see earlier Editori- latent structures (O-PLS) - See http://www.itl.nist.gov/ als 2002). This gives a model relating the The recent O-PLS methods [O-PLS and div898/handbook/pri/section3/ factors to the results, showing which O2-PLS] (see Editorial from April 2002 pri3361.htm for more info. factors are important, and how they on O-PLS), are improved modifications combine in influencing the results. The • Box Behnken designs of the NIPALS PLS algorithm. The devel- model is then used to make predictions, opment of O-PLS has, like the orthogo- - See http://www.itl.nist.gov/ e.g. how to set the factors to achieve div898/handbook/pri/section3/ nal signal correction (OSC) filters desired (optimal) results.The fitted (March 2002 Editorial), been driven by pri3362.htm for more info. model is reviewed by… • D-optimal designs • the large amount of non-correlated Examining the coefficients and variation present in the data sets today, - See http://www.itl.nist.gov/ their 95% confidence interval, or especially in a multivariate calibration div898/handbook/pri/section5/ normal probability plots of ef- situation. The interpretational ability of pri521.htm for more info. fects and interactions. the other inverse regression models

• Examining the ANOVA table, (PLS, PCR, MLR) largely depends on the 2.5 Robustness testing [read more check for curvature degree of systematic orthogonal varia- in Ref 8] • Plotting residuals, normal prob- tion in X with regards to Y. In robustness testing of, for instance, an ability plot of residuals, and run analytical method, the aim is to explore order residuals The basic idea of O2-PLS is to divide the how sensitive the responses are to small • Checking for the optimal trans- systematic part in X and Y into two changes in the factor settings. Ideally, a parts, one which is related to X and Y, robustness test should show that the formation of the response through the use of the Box Cox and one that is not. For each matrix, the responses are not sensitive to small latter is computed in a way that makes it fluctuations in the factors, that is, the plot. • Conclusions: orthogonal to the other matrix, i.e. com- results are the same for all experiments. pletely independent. If X or Y contains 1. Select dominating factors Robustness testing is usually applied as strong but irrelevant variation, O2-PLS 2. Check and modify ranges the last test just before the release of a improves the interpretational ability of 3. Look for curvature product or a method. When performing the parameters in the model, e.g. score a robustness test of a method, the ob- plots, loading plots compared to the jective is 3.1 Multiple • MLR and PLS methods (and other meth- to ascertain that the method is (MLR) ods with similar properties such as ridge robust to small fluctuations in Traditionally, the most frequently used regression). the factor levels, method for finding the regression coeffi- Thus the O2-PLS model can be written • and, if non-robustness is de- cients b is the ordinary as a model, where some tected… method where: factors (T) are common to both X and • to understand how to alter the y=Xb+f Y; T -1 T bounds of the factors so that b=(X X) X y This minimizes the residuals (fTf), which T T robustness may still be claimed. X model: X = TW + TY-orthoPY-ortho +E is equivalent to maximizing the fit to y. T T Y model: Y = UC + UX-orthoPX-ortho + F With MLR, the coefficients of the model T Robustness is achieved when the de- Prediction of Y: Yhat = TC signer understands these potential are computed to minimize the sum of sources of variation and takes steps to the squares of the residuals. In order to 3.4 ANOVA - Analysis of Vari- desensitize the product to them. G. estimate b, MLR requires that the X- ance variables must be linearly independent Taguchi, a Japanese engineer, had a big ANOVA breaks up sums of squares in ((XTX) of full rank). It is important to effect on and experimen- components and compares their size also note that MLR fits one response at tal design in the 1980s and 1990s. The with F-test. a time and hence assumes them to be (see http:// independent. www.stat.rutgers.edu/~buyske/591/ SS_Y = SS_Regression + SS_Residual lect10.pdf for more info.) is a well SS_Residual = known strategy in robustness testing. 3.2 Partial Least Squares Projec- SS_lack_of_fit + SS_pure_error

http://www.acc.umu.se/~tnkjtg/Chemometrics/Editorial Page 5

If center points exist, lack of fit indicates 5. References Further Reading curvature. Lack of fit compares pure • error (from replicated experiments) 1. G.E.P Box, W.G. Hunter and J.S. Introduction to Design of Ex- with residual error (model error). Resid- Hunter "Statistics for experimenters", periments ual plots indicate John Wiley and Sons, Inc,., New York -See http://www.umetrics.com/ (1978) • Outliers pdfs/books/DOEBook.pdf 2. G.E.P Box, N.R. Draper "Empirical • • Curvature Introduction to DoE model-building and Response surfaces", -See http:// • Need for transformations John Wiley and Sons, Inc,., New York kingkong.me.berkeley.edu/ (1987) html/pres_assets/pdfs/ 4. Applications of DoE 3. C.K. Bayne and I.B. Rubin "Practical fracfact2.pdf Experimental Designs and optimizations • DoE overview Engineering Statis- Chemical synthesis methods for chemists", VCH Publishers, tics Handbook (NIST) 1. Synthetic steps Inc., Deerfield Beach, Florida (1986) -See http://www.itl.nist.gov/ 2. Work up and separation 4. Morgan E. - Chemometrics: Experi- 3. Reagents, solvents, catalysts div898/handbook/pri/pri_d.htm mental Design. John Wiley & Sons, Inc., • 4. Structure ' reactivity and proper- New York Optimization designs, NIST ties. 5. Handbook http://www.itl.nist.gov/div898/ (NIST) [http://www.itl.nist.gov/div898/ education/dex/optdesgn/ Biotech industry handbook/index.htm] July 2002. optdesgn.pdf 1. Pharmaceutics, formulation for 6. Carlson R. - Design and Optimization • How to Select Design of Experi- drug delivery in Organic Synthesis. Elsevier science ments Software -See http:// 2. Media development & optimiza- publishers, Amsterdam www.qualitydigest.com/nov98/ tion 7. Slide notes from Svante Wold html/doe.html 3. , drug design (personal communication) • Robust Design and Taguchi 4. Analytical biochemistry, separa- 8. : Principles and Method -See http:// tion (HPLC,…), assay develop- Applications, Umetrics Academy - L. www.stat.rutgers.edu/ ment and optimization Eriksson, E. Johansson, N. Kettaneh- ~buyske/591/lect10.pdf 5. Pharmacology Wold, C. Wikström, and S. Wold, ISBN 6. Process optimization and con- 91-973730-0-1 trol, fermentation, separation, 9. Response Surface Methodology: Proc- purification ess and Product Optimization Using Designed Experiments, 2nd Edition Ray- Cosmetic industry mond H. Myers, Douglas C. Montgom- 1. Processes, production, separa- ery ISBN: 0-471-41255- tion, cleaning,… 2. Formulations, shampoos, nail polish, creams, perfumes, soaps, powders, … 3. Molecular structure, high po- tency, low toxicity, allergenicity

Drug industry 1. Pharmaceutics, formulation for drug release, hardness of pills,… 2. Organic , synthesis, drug design, … 3. , Separation [HPLC, …], resolution, speed. 4. Pharmacology 5. Process optimization and con- trol, synthesis, fermentation, separations, …

Process industry 1. Process optimization and control (yield, purity, through put time, pollution, energy consumption) 2. Product quality and performance (material strength, warp, color, taste, odour) 3. Product stability versus process variation

http://www.acc.umu.se/~tnkjtg/Chemometrics/Editorial Page 6

Appendix Model Parameters statistics, information about model adequacy PRESS, Predicted Residual Sum of Squares:Sum of squared differences between predicted and observed y- values (over all rounds)

Dealing with and quantifying variability http://www.acc.umu.se/~tnkjtg/Chemometrics/Editorial Page 7

Check them out at HoC’s website http:/www.acc.umu.se/~tnkjtg/Chemometrics/Editorial

Stay alert for upcoming editorials on HoC

No Editorial this month July 2002

A 's view of the single-y PLS problem June 2002 Dick Kleinknecht

Wavelets in Chemometrics May 2002 - compression, denoising and feature extraction Johan Trygg

How to create an OSC filter for PLS and end up with a new generic modelling April 2002 method, O-PLS Johan Trygg

Everything you need to know about Orthogonal Signal Correction (OSC) filters March 2002 - and how they can improve interpretation of your data Johan Trygg Have you ever wondered why PLS sometimes needs Feb 2002 more than one component for a single-y vector? Johan Trygg