Thispageintentionallyleftblank Sequential Estimation WILEY SERIES IN PROBABILITY AND STATISTICS Established by WALTER A. SHEWHART and SAMUEL S. WILKS Editors: Vic Barnett, Ralph A. Bradley, Nicholas I. Fisher, J. Stuart Hunter, J. B. Kadane, David G. Kendall, David W. Scott, Adrian F. M. Smith, JozefL. Teugels, Geoffrey S. Watson A complete list of the titles in this series appears at the end of this volume. Sequential Estimation MALAY GHOSH University of Florida NITIS MUKHOPADHYAY University of Connecticut PRANAB K. SEN University of North Carolina A Wiley-Interscience Publication JOHN WILEY & SONS, INC. New York · Chichester · Weinheim · Brisbane · Singapore · Toronto This text is printed on acid-free paper. Copyright © 1997 by John Wiley & Sons, Inc. All rights reserved. Published simultaneously in Canada. Reproduction or translation of any part of this work beyond that permitted by Section 107 or 108 of the 1976 United States Copyright Act without the permission of the copyright owner is unlawful. Requests for permission or farther information should be addressed to the Permissions Department, John Wiley & Sons. Inc.. 605 Third Avenue, New York, NY 10158-0012 Library of Congress Cataloging in Publication Data: Ghosh, Malay. Sequential estimation / Malay Ghosh, Nitis Mukhopadhyay, Pranab K. Sen p. cm. — (Wiley series in probability and statistics. Probability and statistics) Includes bibliographical references (p. - ) and index. ISBN 0-471-81271-4 (cloth : alk. paper) 1. Sequential analysis. 2. Estimation theory. I. Mukhopadhyay, Nitis, 1950- . II. Sen, Pranab Kumar, 1937- . III. Title. QA279.7.G48 1996 519.5'42—dc20 96-32001 10 987654321 Dedicated with affection to Dola Ghosh, Mahua Mukhopadhyay, and Gauri Sen Thispageintentionallyleftblank Contents Preface 1. Introduction and Coverage 1.1 Introduction, 1 1.2 Some Sequential Sampling Schemes in Practice, 7 1.2.1 Binomial Waiting-Time Distribution, 8 1.2.2 Hypergeometric Waiting-Time Distribution, 8 1.2.3 Capture-Mark-Recapture Procedures, 9 1.2.4 Time-Sequential Models, 10 1.2.5 Sequential Models in Reliability Problems, 11 1.2.6 Recursive Estimation and Sequential Schemes, 12 1.3 Organization of This Book, 12 2. Probabilistic Results in Sequential Analysis 2.1 Introduction, 19 2.2 Martingales, 19 2.3 Stopping Times, 21 2.4 Martingale Inequalities and Identities, 24 2.5 Submartingale Convergence Theorems, 35 2.6 Martingale Central Limit Theorems, 40 2.7 Random Central Limit Theorems and Berry-Esseen Bounds, 2.8 Renewal Theorem—First Passage and Residual Waiting Times, 2.9 Nonlinear Renewal Theory, 58 2.10 Exercises, 65 3. Some Basic Concepts for Fixed-Sample Estimation 3.1 Introduction, 69 3.2 Decision-Theoretic Notions, 69 VIH CONTENTS 3.3 Bayesian Decision Rules, 73 3.4 Sufficiency and Efficiency, 75 3.5 Invariance and Transitivity, 81 3.6 Method of Maximum Likelihood, 82 3.7 Why Sequential? 84 3.8 Exercises, 85 4. General Aspects of Sequential Estimation 89 4.1 Introduction, 89 4.2 Sufficiency, Rao-Blackwell Theorem, and Transitivity, 90 4.3 Cramér-Rao and Related Inequalities, 96 4.4 Sequential Binomial Sampling Plans, 101 4.5 Exercises, 107 5. Sequential Bayesian Estimation 111 5.1 Introduction. Ill 5.2 Bayesian Sequential Decision Rules, 112 5.3 Sequential Bayesian Estimation, 122 5.4 Asymptotically Pointwise Optimal (APO) Stopping Rules, 125 5.5 Hierarchical and Empirical Bayes Sequential Estimation, 138 5.6 Exercises, 150 6. Multistage Estimation 6.1 Introduction, 153 6.2 Fixed-Width Confidence Intervals and Two-Stage Procedures, 6.2.1 Stein's Two-Stage Procedure, 154 6.2.2 Modified Two-Stage Procedure, 156 6.2.3 Further Generalizations, 157 6.3 Fixed-Width Confidence Intervals and Three-Stage Procedures, 6.3.1 The Global Theory, 160 6.3.2 Applications of the Three-Stage Procedure, 164 6.4 Fixed-Width Confidence Intervals and Accelerated Sequential Procedures, 168 6.4.1 The Global Theory, 169 6.5 Point Estimation Problems, 173 6.5.1 Minimum Risk Normal Mean Problem, 173 6.5.2 Two-Stage Procedure, 174 6.5.3 Modified Two-Stage Procedure, 175 6.5.4 Three-Stage Procedure, 175 CONTENTS u 6.5.5 Accelerated Sequential Procedure, 177 6.6 Other Related Estimation Problems, 178 6.6.1 Point Estimation in Exponential Populations, 178 6.6.2 Estimation of Normal Variance, 182 6.6.3 Binomial and Negative Binomial Problems, 184 6.7 Comparison of Populations, 185 6.7.1 Fixed-Width Confidence Intervals, 185 6.7.2 Point Estimation, 188 6.8 Estimation in Multivariate Normal and Linear Models, 191 6.8.1 Estimation of Mean Vector When Σ Is Arbitrary, 192 6.8.2 Comparison of Populations, 197 6.8.3 Linear Regression Problems, 197 6.8.4 Shrinkage Estimators, 202 6.8.5 Estimation of Ordered Parameters, 203 6.9 Exercises, 204 7. Parametric Sequential Point Estimation 211 7.1 Introduction, 211 7.2 Estimation of the Normal Mean, 212 7.3 Estimation of the Difference of Two Normal Means, 222 7.4 Point Estimation in Linear Models, 224 7.5 Estimation of the Multivariate Normal Mean, 227 7.6 Sequential Shrinkage Estimation, 232 7.7 Sequential Estimation of the Gamma Scale Parameter, 240 7.8 Exercises, 243 8. Parametric Sequential Confidence Estimation 249 8.1 Introduction, 249 8.2 Fixed-Width Interval Estimation of the Normal Mean, 249 8.3 Sequential Interval Estimation of the Difference of Two Normal Means, 256 8.4 Fixed-Size Confidence Bounds for Linear Regression Parameters, 260 8.5 Confidence Region for the Mean Vector, 263 8.6 Exercises, 265 9. Nonparametric Sequential Point Estimation 269 9.1 Introduction, 269 9.2 Estimable Parameters and MRE, 270 9.3 Differentiable Statistical Functionals and MRE, 287 X CONTENTS 9.4 Simple Semiparametric Models, 293 9.5 Multiparameter AMRE, I, 303 9.6 Multiparameter AMRE, II, 309 9.7 Exercises, 312 10. Nonparametric Sequential Confidence Estimation 315 10.1 Introduction, 315 10.2 Type-A Confidence Intervals, 316 10.3 Type-B Confidence Intervals, 323 10.4 Nonparametric Confidence Sets, 328 10.5 Exercises, 332 11. Estimation Following Sequential Tests 335 11.1 Introduction, 335 11.2 Bias and Confidence Interval Evaluations, 335 11.2.1 Unknown Variance Case, 338 11.2.2 Another Practical Approach, 339 11.3 Sequential χ2 and F Tests, 340 11.4 Exercises, 341 12. Time-Sequential Estimation Problems 343 12.1 Introduction, 343 12.2 Time-Sequential Estimation for Poisson and Wiener Processes, 345 12.3 Time-Sequential Estimation for Exponential Life-Testing Models, 350 12.4 Some Generalizations, 359 12.5 Exercises, 364 13. Sequential Estimation in Reliability Models 367 13.1 Introduction, 367 13.2 Bundle Strength of Filaments, 368 13.3 System Reliability and Availability, 377 13.4 Sequential Estimation of Functional Parameters, 383 13.5 Exercises, 390 14. Sequential Estimation of the Size of a Finite Population 393 14.1 Introduction, 393 14.2 The CMRR and Two-Sample Estimators of N, 394 CONTENTS 14.3 The CMRR and Multisample Estimators of N, 397 14.4 Estimation of N Under Inverse Sampling Schemes, 405 14.5 Sequential Tagging Schemes, 407 14.6 Bounded Percentage Width Confidence Interval for N, 412 14.7 Asymptotically Optimal Sequential Point Estimation of N, 4 14.8 Exercises, 421 15. Stochastic Approximation 15.1 Introduction, 425 15.2 General Asymptotics, 426 15.3 Sequential Perspectives. 431 15.4 Exercises, 443 References Author Index Subject Index Thispageintentionallyleftblank Preface Sequential analysis has made great advances since its inception in the United States and United Kingdom during the Second World War. Its success can be attributed in part to the development of sophisticated probabilistic and inferential techniques that have enriched statistics in general, but much of it is due to its varied applications such as clinical trials, quality technology, and reliability engineering, to name a few. The total coverage of sequential analysis is indeed so huge that it is even beyond the capability of an encyclopedic volume. Among the different topics, the one that has received the greatest attention is sequential hypothesis testing. Wald's (1947) seminal book contains its early development in the 1940s. The development of the next two decades is mirrored admirably in Ghosh (1970). More recent theoretical development appears in Siegmund (1985). In contrast, sequential estimation has received scant attention, a notable excep- tion being Govindarajulu (1981), where an attempt has been made to combine se- quential hypothesis testing and estimation problems in a single volume, albeit re- sulting in some lack of uniformity and clarity of comprehension. Sequential nonparametrics treated in Sen (1981 a) contains some account of sequential estima- tion, though primarily in the context of nonparametric location and regression mod- els, while the Handbook of Sequential Analysis (Ghosh and Sen, 1991) contains several chapters devoted to sequential estimation, albeit in an application-oriented fashion. However, significant advances have been made over the past 15 years, the most noteworthy work being in the area of three-stage accelerated sequential sampling procedures and more recently in related nonparametric and semiparametric sequen- tial estimation procedures. However, these advances are not fully captured in any text, and there is a profound need to tie up the diversities in sequential estimation in a logically integrated and unified manner. The focus of our book is sequential estimation. It treats both classical and mod- ern techniques. Moreover it includes both parametric and nonparametric methods. Among some of the topics not properly included in other contemporary texts, we mention shrinkage, empirical and hierarchical Bayes procedures, time-sequential estimation, empirical and hierarchical populations sampling, reliability estimations, and capture-recapture methodology leading to sequential schemes. Xlll XIV PREFACE The book is primarily intended for researchers in sequential analysis, but it can also be used as a special topics course for advanced graduate students.