Stochastic Processes and Models This page intentionally left blank Stochastic Processes and Models David Stirzaker St John’s College, Oxford 1 3 Great Clarendon Street, Oxford OX2 6DP Oxford University Press is a department of the University of Oxford. It furthers the University’s objective of excellence in research, scholarship, and education by publishing worldwide in Oxford New York Auckland Cape Town Dar es Salaam Hong Kong Karachi Kuala Lumpur Madrid Melbourne Mexico City Nairobi New Delhi Shanghai Taipei Toronto With offices in Argentina Austria Brazil Chile Czech Republic France Greece Guatemala Hungary Italy Japan Poland Portugal Singapore South Korea Switzerland Thailand Turkey Ukraine Vietnam Oxford is a registered trade mark of Oxford University Press in the UK and in certain other countries Published in the United States by Oxford University Press Inc., New York © Oxford University Press, 2005 The moral rights of the author have been asserted Database right Oxford University Press (maker) First published 2005 All rights reserved. No part of this publication may be reproduced, stored in a retrieval system, or transmitted, in any form or by any means, without the prior permission in writing of Oxford University Press, or as expressly permitted by law, or under terms agreed with the appropriate reprographics rights organization. Enquiries concerning reproduction outside the scope of the above should be sent to the Rights Department, Oxford University Press, at the address above You must not circulate this book in any other binding or cover and you must impose the same condition on any acquirer British Library Cataloguing in Publication Data Data available Library of Congress Cataloging in Publication Data Data available Typeset by Newgen Imaging Systems (P) Ltd., Chennai, India Printed in Great Britain on acid-free paper by Biddles Ltd., King’s Lynn, Norfolk ISBN 0–19–856813–4 978–0–19–856813–1 ISBN 0–19–856814–2(Pbk.) 978–0–19–856814–8(Pbk.) 13579108642 Contents Preface ix 1 Probability and random variables 1 1.1 Probability 1 1.2 Conditional probability and independence 4 1.3 Random variables 6 1.4 Random vectors 12 1.5 Transformations of random variables 16 1.6 Expectation and moments 22 1.7 Conditioning 27 1.8 Generating functions 33 1.9 Multivariate normal 37 2 Introduction to stochastic processes 45 2.1 Preamble 45 2.2 Essential examples; random walks 49 2.3 The long run 56 2.4 Martingales 63 2.5 Poisson processes 71 2.6 Renewals 76 2.7 Branching processes 87 2.8 Miscellaneous models 94 2.9 Some technical details 101 3 Markov chains 107 3.1 The Markov property; examples 107 3.2 Structure and n-step probabilities 116 3.3 First-step analysis and hitting times 121 3.4 The Markov property revisited 127 3.5 Classes and decomposition 132 3.6 Stationary distribution: the long run 135 3.7 Reversible chains 147 vi Contents 3.8 Simulation and Monte Carlo 151 3.9 Applications 157 4 Markov chains in continuous time 169 4.1 Introduction and examples 169 4.2 Forward and backward equations 176 4.3 Birth processes: explosions and minimality 186 4.4 Recurrence and transience 191 4.5 Hitting and visiting 194 4.6 Stationary distributions and the long run 196 4.7 Reversibility 202 4.8 Queues 205 4.9 Miscellaneous models 209 5 Diffusions 219 5.1 Introduction: Brownian motion 219 5.2 The Wiener process 228 5.3 Reflection principle; first-passage times 235 5.4 Functions of diffusions 242 5.5 Martingale methods 250 5.6 Stochastic calculus: introduction 256 5.7 The stochastic integral 261 5.8 Itô’s formula 267 5.9 Processes in space 273 6 Hints and solutions for starred exercises and problems 297 Further reading 323 Index 325 ...and chance, though restrained in its play within the right lines of necessity, and sideways in its motions directed by free will, though thus prescribed to by both, chance by turns rules either, and has the last featuring blow at events. Herman Melville, Moby Dick In such a game as this there is no real action until a guy is out on a point, and then the guys around commence to bet he makes this point, or that he does not make this point, and the odds in any country in the world that a guy does not make a ten with a pair of dice before he rolls seven, is 2 to 1. Damon Runyon, Blood Pressure This page intentionally left blank Preface This book provides a concise introduction to simple stochastic processes and models, for readers who have a basic familiarity with the ideas of ele- mentary probability. The first chapter gives a brief synopsis of the essential facts usually covered in a first course on probability, for convenience and ease of reference. The rest of the text concentrates on stochastic processes, developing the key concepts and tools used in mainstream applications and stochastic models. In particular we discuss: random walks, renewals, Markov chains, martingales, the Wiener process model for Brownian motion, and diffusion processes. The book includes examples and exercises drawn from many branches of applied probability. The principal purpose here is to introduce the main ideas, applications and methods of stochastic modelling and problem-solving as simply and compactly as possible. For this reason, we do not always display the full proof, or most powerful version, of the underlying theorems that we use. Frequently, a simpler version is substituted if it illustrates the essential ideas clearly. Furthermore, the book is elementary, in the sense that we do not give any rigorous development of the theory of stochastic processes based on measure theory. Each section ends with a small number of exercises that should be tackled as a matter of routine. The problems at the end of any chapter have the potential to require more effort in their solution. The ends of proofs, examples, and definitions are denoted by the symbols , , and , respectively. And all these, with key equations, are numbered sequentially by a triple of the form (1.2.3); this example refers to the third result in the second section of Chapter 1. Within sections a single number is used to refer to an equation or proof etc. in the same section. Exercises are lettered (a), (b), (c), and so on, whereas Problems are numbered; solutions to selected Exercises and Problems appear at the end of the book, and these are starred in the text. This page intentionally left blank Probability and random 1 variables To many persons the mention of Probability suggests little else than the notion of a set of rules, very ingenious and profound rules no doubt, with which mathematicians amuse themselves by setting and solving puzzles. John Venn, The Logic of Chance “It says here he undoubtedly dies of heart disease. He is 63 years old and at this age the price is logically thirty to one that a doctor will die of heart disease. Of course,” Ambrose says, “if it is known that a doctor of 63 is engaging in the rumba, the price is one hundred to one.” Damon Runyon, Broadway Incident This chapter records the important and useful results that are usually included in introductory courses on probability. The material is presented compactly for purposes of reference, with long proofs and elaborations mostly omitted. You may either skim the chapter briskly to refresh your memory, or skip it until you need to refer back to something. 1.1 Probability Randomness and probability are not easy to define precisely, but we certainly recognize random events when we meet them. For example, ran- domness is in effect when we flip a coin, buy a lottery ticket, run a horse race, buy stocks, join a queue, elect a leader, or make a decision. It is nat- ural to ask for the probability that your horse or candidate wins, or your stocks rise, or your queue quickly shrinks, and so on. In almost all of these examples the eventualities of particular importance to you form a proportion of the generality; thus: a proportion of all stocks rise, a proportion of pockets in the roulette wheel let your bet win, a pro- portion of the dartboard yields a high score, a proportion of voters share your views, and so forth. In building a model for probability therefore, we have it in mind as an extension of the idea of proportion, formulated in a more general framework. After some reflection, the following definitions and rules are naturally appealing. 2 1 : Probability and random variables Note that, here and elsewhere, a word or phrase in italics is being defined. (1) Experiment. Any activity or procedure that may give rise to a well- defined set of outcomes is called an experiment. The term well-defined means simply that you can describe in advance everything that might happen. (2) Sample space. The set of all possible outcomes of an experiment is called the sample space, and is denoted by . A particular but unspecified outcome in may be denoted by ω. The main purpose of a theory of probability is to discuss how likely vari- ous outcomes of interest may be, both individually and collectively, so we define: (3) Events.Anevent is a subset of the sample space . In particular, is called the certain event, and its complement c is an event called the impossible event, which we denote by c =∅, the empty set. If two events A and B satisfy A ∩ B =∅, then they are said to be disjoint. Note that while all events are subsets of , not all subsets of are neces- sarily events.