University of Tennessee, Knoxville TRACE: Tennessee Research and Creative Exchange Doctoral Dissertations Graduate School 12-2003 Prediction Interval Estimation Techniques for Empirical Modeling Strategies and their Applications to Signal Validation Tasks Brandon Peter Rasmussen University of Tennessee - Knoxville Follow this and additional works at: https://trace.tennessee.edu/utk_graddiss Part of the Nuclear Engineering Commons Recommended Citation Rasmussen, Brandon Peter, "Prediction Interval Estimation Techniques for Empirical Modeling Strategies and their Applications to Signal Validation Tasks. " PhD diss., University of Tennessee, 2003. https://trace.tennessee.edu/utk_graddiss/2379 This Dissertation is brought to you for free and open access by the Graduate School at TRACE: Tennessee Research and Creative Exchange. It has been accepted for inclusion in Doctoral Dissertations by an authorized administrator of TRACE: Tennessee Research and Creative Exchange. For more information, please contact [email protected]. To the Graduate Council: I am submitting herewith a dissertation written by Brandon Peter Rasmussen entitled "Prediction Interval Estimation Techniques for Empirical Modeling Strategies and their Applications to Signal Validation Tasks." I have examined the final electronic copy of this dissertation for form and content and recommend that it be accepted in partial fulfillment of the requirements for the degree of Doctor of Philosophy, with a major in Nuclear Engineering. J. Wesley Hines, Major Professor We have read this dissertation and recommend its acceptance: Robert E. Uhrig, Belle R. Upadhyaya, Hamparsum Bozdogan, Andrei Gribok Accepted for the Council: Carolyn R. Hodges Vice Provost and Dean of the Graduate School (Original signatures are on file with official studentecor r ds.) To the Graduate Council: I am submitting herewith a dissertation written by Brandon Peter Rasmussen entitled "Prediction Interval Estimation Techniques for Empirical Modeling Strategies and their Applications to Signal Validation Tasks." I have examined the final electronic copy of this dissertation for form and content and recommend that it be accepted in partial fulfillment of the requirements for the degree of Doctor of Philosophy, with a major in Nuclear Engineering. J. Wesley Hines______________ Major Professor We have read this dissertation and recommend its acceptance: Robert E. Uhrig___________________ Belle R. Upadhyaya________________ Hamparsum Bozdogan______________ Andrei Gribok____________________ Accepted for the Council: Anne Mayhew_______________ Vice Provost and Dean of Graduate Studies (Original signatures are on file with official student records.) Prediction Interval Estimation Techniques for Empirical Modeling Strategies and their Applications to Signal Validation Tasks A Dissertation Presented for the Doctor of Philosophy Degree The University of Tennessee, Knoxville Brandon P. Rasmussen December 2003 Copyright © 2003 by Brandon P. Rasmussen All rights reserved. ii DEDICATION for Shasten iii ACKNOWLEDGEMENTS I would like to thank the members of my dissertation committee. Dr. J. Wesley Hines, my major professor, who guided me through my years of graduate study. Dr. Hines is an excellent and dedicated professional and it has been a privilege to work with him these past 6 years. Dr. Robert E. Uhrig, who has taken the time to review my work and provide guidance based on his considerable experience in both artificial intelligence and the nuclear power industry. Dr. Belle R. Upadhyaya, who has taken the time to review this work and has always been ready and willing to help out along the way. Dr. Hamparsum Bozdogan, who has brought his expertise in statistics to the table in reviewing and guiding this work. Dr. Andrei Gribok, who has provided the basis for portions of this dissertation through his work on regularization and nonparametric regression analysis. While three members of the Nuclear Engineering department's faculty were on my dissertation committee, Dr. J. Wesley Hines, Dr. Belle R. Upadhyaya, and Dr. Robert E. Uhrig, I would also like to thank the rest of the faculty members. Dr. Lawrence W. Townsend, Dr. Ronald E. Pevey, Dr. Peter G. Groer, Dr. Arthur E. Ruggles, and Dr. Laurence F. Miller, all of whom are among the finest professors I have had the pleasure of studying under. I would like to recognize the efforts of our department head Dr. H. Lee Dodds. His assistance and guidance over the course of my graduate studies has been a valuable asset, and his dedication to his work is commendable. I would especially like to thank the organizations and companies that provided the funding for my studies and the completion of this work, the Department of Energy under a Nuclear Engineering Fellowship, Smart Signal, Lisle, Illinois, and the Institutt for Energiteknikk, Halden, Norway. Finally, I would like to thank the staff of the Nuclear Engineering department who always provided any assistance that I requested. Our computer gurus Gary Graves and iv Dick Bailey always knew what to do and how to fix it. Kristin England, Ellen Fisher, Lydia Salmon, and Juvy Melton never let me get by without the necessary paperwork, and were always ready to provide their assistance. v ABSTRACT The basis of this work was to evaluate both parametric and non-parametric empirical modeling strategies applied to signal validation or on-line monitoring tasks. On-line monitoring methods assess signal channel performance to aid in making instrument calibration decisions, enabling the use of condition-based calibration schedules. The three non-linear empirical modeling strategies studied were: artificial neural networks (ANN), neural network partial least squares (NNPLS), and local polynomial regression (LPR). These three types are the most common nonlinear models for applications to signal validation tasks. Of the class of local polynomials (for LPR), two were studied in this work: zero-order (kernel regression), and first-order (local linear regression). The evaluation of the empirical modeling strategies includes the presentation and derivation of prediction intervals for each of three different model types studied so that estimations could be made with an associated prediction interval. An estimate and its corresponding prediction interval contain the measurements with a specified certainty, usually 95%. The prediction interval estimates were compared to results obtained from bootstrapping via Monte Carlo resampling, to validate their expected accuracy. The estimation of prediction intervals applied to on-line monitoring systems is essential if widespread use of these empirical based systems is to be attained. In response to the topical report "On-Line Monitoring of Instrument Channel Performance," published by the Electric Power Research Institute [Davis 1998], the NRC issued a safety evaluation report that identified the need to evaluate the associated uncertainty of empirical model estimations from all contributing sources. This need forms the basis for the research completed and reported in this dissertation. The focus of this work, and basis of its original contributions, were to provide an accurate prediction interval estimation method for each of the mentioned empirical modeling techniques, and to verify the results via bootstrap simulation studies. Properly determined prediction interval estimates were obtained that consistently captured the vi uncertainty of the given model such that the level of certainty of the intervals closely matched the observed level of coverage of the prediction intervals over the measured values. In most cases the expected level of coverage of the measured values within the prediction intervals was 95%, such that the probability that an estimate and its associated prediction interval contain the corresponding measured observation was 95%. The results also indicate that instrument channel drifts are identifiable through the use of the developed prediction intervals by observing the drop in the level of coverage of the prediction intervals to relatively low values, e.g. 30%. While all empirical models exhibit optimal performance for a given set of specifications, the identification of this optimal set may be difficult to attain. The developed methods of prediction interval estimation were shown to perform as expected over a wide range of model specifications, including misspecification. Model misspecification occurs through different mechanisms dependent on the type of empirical model. The main mechanisms under which model misspecification occur for each empirical model studied are: ANN – through architecture selection, NNPLS – through latent variable selection, LPR – through bandwidth selection. In addition, all of the above empirical models are susceptible to misspecification due to inadequate data and the presence of erroneous predictor variables in the set of predictors. A study was completed to verify that the presence of erroneous variables, i.e. unrelated to the desired response or random noise components, resulted in increases in the prediction interval magnitudes while maintaining the appropriate level of coverage for the response measureme nts. In addition to considering the resultant prediction intervals and coverage values, a comparative evaluation of the different empirical models was performed. The evaluation considers the average estimation errors and the stability of the models under repeated Monte Carlo resampling. The results indicate the large uncertainty of ANN models applied to collinear