System Regularities in Design of Experiments and Their Application by Xiang Li B.S. and M.S., Engineering Mechanics Tsinghua University, 2000, 2002 Submitted to the Department of Mechanical Engineering in Partial Fulfillment of the Requirements for the Degree of Doctor of Philosophy in Mechanical Engineering at the Massachusetts Institute of Technology June, 2006 © 2006 Massachusetts Institute of Technology. All rights reserved. Signature of Author ……………………………………………………………….... Department of Mechanical Engineering May 9, 2006 Certified by .………………………………………………………………………… Daniel D. Frey Assistant Professor of Mechanical Engineering and Engineering Systems Thesis Supervisor Accepted by ..……………………………………………………………………….. Lallit Anand Chairman, Department Committee on Graduate Students System Regularities in Design of Experiments and Their Application by Xiang Li Submitted to the Department of Mechanical Engineering on May 9, 2006, in partial fulfillment of the requirements for the Degree of Doctor of Philosophy in Mechanical Engineering ABSTRACT This dissertation documents a meta-analysis of 113 data sets from published factorial experiments. The study quantifies regularities observed among main effects and multi-factor interactions. Such regularities are critical to efficient planning and analysis of experiments, and to robust design of engineering systems. Three previously observed properties are analyzed – effect sparsity, hierarchy, and heredity. A new regularity on effect synergism is introduced and shown to be statistically significant. It is shown that a preponderance of active two-factor interaction effects are synergistic, meaning that when main effects are used to increase the system response, the interactions provide an additional increase and that when main effects are used to decrease the response, the interactions generally counteract the main effects. Based on the investigation of system regularities, a new strategy is proposed for evaluating and comparing the effectiveness of robust parameter design methods. A hierarchical probability model is used to capture assumptions about robust design scenarios. A process is presented employing this model to evaluate robust design methods. This process is then used to explore three topics of debate in robust design: 1) the relative effectiveness of crossed versus combined arrays; 2) the comparative advantages of signal-to-noise ratios versus response modeling for analysis of crossed arrays; and 3) the use of adaptive versus “one shot” methods for robust design. For the particular scenarios studied, it is shown that crossed arrays are preferred to combined arrays regardless of the criterion used in selection of the combined array. It is shown that when analyzing the data from crossed arrays, signal-to-noise ratios generally provide superior performance; although that response modeling should be used when three-factor interactions are absent. Most significantly, it is shown that using an adaptive inner array design crossed with an orthogonal outer array resulted in far more improvement on average than other alternatives. Thesis Supervisor: Daniel D. Frey Title: Assistant Professor of Mechanical Engineering and Engineering Systems 2 Acknowledgments I would like to thank a number of people who contributed in many ways in making this thesis possible. First, I am grateful to my advisor, Professor Daniel D. Frey, for giving me the opportunity to work on such a challenging topic. With deep research insights and rich engineering knowledge, he has patiently guided me to explore the complicated topic of effectively applying experimental designs to engineering systems and to conquer a variety of difficulties along the way. In addition, he always gave me the freedom to pursue my interests and always offered me great opportunities to widen my perspective. I also wish to thank my doctoral committee members, Professor Warren P. Seering and Professor Roy E. Welsch, for their invaluable advice and inspiring discussions on this research. Thanks to the staff members of MIT who have been in great assistance to this work. I would especially like to thank Ms. Leslie Regan, Ms. Joan Kravit, Ms. Maggie Sullivan, Mr. Jason Pring, and Ms. Danielle Guichard-Ashbrook. I also owe a debt of gratitude to Mr. Steven Schondorf for being my mentor and giving me continuous support. I would like to sincerely acknowledge the encouragement from my colleagues in the Robust Design Group: Rajesh Jugulum, Yiben Lin, Hungjen Wang, Chad Forster, Nandan Sudarsanam, Danielle Zurovcik, Jagmeet Singh, and Troy Savoie, who have been wonderful friends and research partners. Finally, I would like to express my deepest thanks to my parents and my sister for their love and encouragement. This work could not have been done without their support. In addition, my close friends have always kept me sane and motivated: Elaine, Chen, Shuang, Dave, Bolero, Huajie, and Hao – thanks for everything. 3 Contents 1. Introduction..........................................................................................................11 1.1 Overview......................................................................................................................... 11 1.2 Motivation....................................................................................................................... 13 1.2.1 Debate on Adaptive Experimentation...................................................................... 14 1.2.2 Debate on Robust Design Methods.......................................................................... 15 1.3 Research Objectives........................................................................................................ 16 1.4 Research Roadmap.......................................................................................................... 17 1.5 Organization of the Dissertation ..................................................................................... 19 2. Regularities in Data from Experiments ...............................................................21 2.1 Overview......................................................................................................................... 21 2.2 Design of Experiments.................................................................................................... 22 2.2.1 A Historical Review................................................................................................. 23 2.2.2 A Technical Review................................................................................................. 24 2.2.3 Adaptive One-Factor-at-A-Time (OFAT) Experiments .......................................... 27 2.3 Introduction to System Regularities................................................................................ 29 2.3.1 Effect Sparsity.......................................................................................................... 30 2.3.2 Effect Hierarchy....................................................................................................... 33 2.3.3 Effect Heredity......................................................................................................... 35 2.4 DOE Models Incorporating System Regularities ........................................................... 36 2.4.1 The General Linear Model....................................................................................... 36 2.4.2 The Relaxed Weak Heredity Model ........................................................................ 37 2.4.3 The Hierarchical Probability Model ........................................................................ 39 2.5 Effects of System Regularities on DOE Methods........................................................... 42 2.6 Summary......................................................................................................................... 49 3. Verification and Quantification of System Regularities......................................50 3.1 Overview......................................................................................................................... 50 3.2 Objectives and Methods.................................................................................................. 51 3.2.1 The General Linear Model Revisited....................................................................... 52 3.2.2 The Lenth Method for Effect Analysis .................................................................... 52 3.2.3 Method for Quantifying Effect Sparsity .................................................................. 56 3.2.4 Method for Quantifying Effect Hierarchy ............................................................... 57 3.2.5 Method for Quantifying Effect heredity .................................................................. 58 4 3.3 The Set of Experimental Data......................................................................................... 59 3.4 An Illustrative Example for a Single Data Set................................................................ 62 3.5 Results of Meta-Analysis of 133 Data Sets .................................................................... 68 3.6 Quantification of the Standard Deviation c .................................................................... 71 3.7 Conclusions and Discussion ........................................................................................... 73 3.8 Summary........................................................................................................................