Decaying Time of Skip Free Markov Chains

by

Garimella Ramamurthy

Report No: IIIT/TR/2016/-1

Centre for Security, Theory and Algorithms International Institute of Information Technology Hyderabad - 500 032, INDIA October 2016 Decaying Time of Skip Free Markov Chains

Rama Murthy Garimella Rumyantsev Alexander, SPCRC, Institute of Applied Mathematical Research, IIIT, Hyderabad, India Karelian Research Center, Russia

ABSTRACT

In this research paper, the concept of “Decaying Time” of a Skip Free Markov chain is introduced. Bounds to estimate decaying time of Quasi-Birth-and-Death process ( QBD) as well as G/M/1-type process are derived. These bounds are utilized in approximating the equilibrium probability distribution of Skip Free Markov Chains.

1. Introduction: Markov chains provide interesting stochastic models of various natural/artificial phenomena. Specifically, Continuous Time Markov Chains (CTMCs) are utilized extensively in queueing theory and its applications. These stochastic processes are studied extensively since they exhibit equilibrium behavior. Thus, efficient computation of equilibrium probability distribution of CTMC’s is considered as an ever green open research problem. A class of two dimensional Markov chains, called Quasi- Birth-and-Death processes exhibit a matrix geometric solution for the equilibrium probability distribution. Consider a Quasi-Birth-and-Death ( QBD) process i.e. two dimensional Continuous Time Markov Chain with state space, E i.e.

The generator matrix of such a CTMC is of the following form:

Thus, the equilibrium probability vector is partitioned in the following manner: . For such Continuous Time Markov Chains, we have that

the

We call such a solution as Marcel Neuts generalized the result by showing that G/M/1-type Markov chains also exhibit matrix geometric solution for the equilibrium Probability Mass Function ( PMF ). Markov chains of G/M/1- type as well as M/G/1-type are called Skip Free Markov chains because of the structure of the generator matrix. In the case of G/M/1-type Markov chains, the recursion matrix , called RATE MATRIX is the minimal nonnegative solution of the matrix power series equation i.e. where, the matrices are the sub-matrices in the generator matrix of G/M/1-type Markov chain. There are mainly two approaches to compute the solution i.e. rate matrix, R: (i) Iterative Procedures, (ii) Closed form method i.e. computation of the Jordan Canonical Form representation of rate matrix, R. In approach (i), the matrix power series equation is truncated and a high degree matrix polunomial equation is utilized. The CRITERIA FOR TRUNCATION has been very ADHOC in nature ( without rigorous justification ). In this research paper, we propose a FORMAL CRITERIA for truncation of the matrix power series. We formulate the concept of DECAYING TIME of a skip free Markov chain. This concept is SIMILAR in the spirit of MIXING TIME of a Markov chain. This research paper is organized as follows. In Section 2, related literature is reviewed. In Section 3, the concept of decaying time is formally discussed. The relationship of decaying time with the smallest and largest eigenvalues of rate matrix is discussed. In Section 4, an interesting upper bound on the smallest eigenvalues of rate matrix of a QBD process is derived. The bound is naturally associated with the decaying time of QBD process. The derivation of upper bound is generalized to the rate matrix of a G/M/1-type Markov process in Section 5. The research paper concludes in Section 6.

2. Related Literature: Efficient and numerically stable computation of rate matrix of a skip free Markov chain such as G/M/1-type Markov process is an interesting research problem. There are mainly two approaches for computing the rate matrix: (I) Iterative Procedures and (II) Closed form Method. Several researchers proposed iterative procedures to compute the rate matrix claiming that such procedures are numerically stable. But in such procedures matrix polynomial/matrix power series is truncated and a lower degree matrix polynomial is utilized.

3. Decaying Time Concept and its Interpretation: The concept of decaying time is motivated by the well known concept of We first explain the mixing time concept in the context of Discrete Time Markov Chains ( DTMCs). It is well known that the spectral radius of state transition matrix of a DTMC is ONE. Thus, the magnitude of second largest eigenvalue determines, how fast the transient probability distribution of associated DTMC reaches the equilibrium probability distribution. The

eigenvalue to decay to zero. From the matrix geometric solution of a Skip- Free Markov Chain, specifically G/M/1-type Markov chain, we have that

Hence, it is clear that the equilibrium probability vector of states on level “M” approaches zero as M tends to infinity ( since the spectral radius of rate matrix, R is strictly less than zero ). Such a formal concept enables truncating the matrix power series equation into matrix polynomial equation ( as done by other researchers ADHOCly to compute the rate matrix by iterative procedures ). We formally define the concept below.

Definition: Decaying time of G/M/1-type Markov Chain is with being a column vector of all ones. It is well known that the rate matrix, R is irreducible, sub-stochastic and nonnegative. Hence, by Perron-Frobenius Theorem, the spectral radius of rate matrix, R i.e. is positive, real and simple ( i.e. of multiplicity one). Thus,

Also, if denotes the smallest eigenvalue of R ( which could be complex valued ), then

Thus,

(I) determines how slowly the sum of probabilities of states on a level approach zero i.e. it is associated with slowly decaying mode of the rate matrix, R.

(II) In the same spirit, determines how fast the sum of probabilities of states on a level approach zero i.e. it is associated with fast decaying mode of the rate matrix, R.

Traditionally, researchers utilized the spectral radius of R i.e. for truncating the matrix power series/ matrix polynomial ( in the case of G/M/1- type Markov chain ), as it was known how to bound the spectral radius of R. It was not realized that the smallest eigenvalue of rate matrix R can bounded when R is non-singular. We readily realize that is related to the upper bound on the smallest eigenvalue of rate matrix R, when R is non-singular.

We are thus, naturally motivated to bound and we succeeded in finding an upper bound discussed in Section 4.

Note: In the case of iterative methods for the computation of rate matrix, R ( which are claimed to be numerically stable ), the eigenvalues of rate matrix, R are not explicitly computed ( as in the case of closed form method ). Thus computable bounds on spectral radius of R ( as well as the smallest eigenvalue ) are needed to truncate the matrix power serie / matrix polynomial ( associated with G/M/1-type Markov Chain ) based on formal reasoning. It is thus clear that the bounds must be specified in terms of the sub-matrices of the generator matrix i. e

4. Bounds on Decaying Time of QBD Process: In the literature on QBD processes, there are results on bounding the spectral radius of rate matrix, R. But, to the best of our knowledge, there are no results on bounding the smallest eigenvalue of R, when it is non-singular. It can be readily seen that

Thus, we would like to bound the smallest eigenvalue of R when the matrix is non-singular. We need the following lemma in arriving at the required bound.

Lemma 1: is a stochastic matrix

Proof: The following factorization lemma is well known i.e.

Let be a column vector of all ones. Since the matrices are sub-matrices in the generator of QBD process, we readily have that

Thus, is a generator matrix. Further, since all the eigenvalues of rate matrix, R are all strictly inside the unit circle, we have that Equivalently,

For a positive recurrent QBD process, since

is a diagonally dominant matrix and hence is non-singular.

It can be easily seen that the inverse of ( diagonally dominant matrix with negative diagonal elements and non-negative off diagonal elements ) is a NON-POSITIVE matrix. Thus, we readily have that .

Hence, it readily follows that is a stochastic matrix. Q.E.D. Note: We are interested in deriving computable bound on in terms of the elements of the matrices

From the above Lemma, we have that

is a stochastic matrix. Thus

Hence

Since, P is stochastic matrix, we have that

Hence

Equivalently, ……………………….(1)

Further, from the matrix quadratic equation, we have that

Thus

Since, R is sub-stochastic with smallest eigenvalue we have that

Also,

Thus,

Hence, we have

.

Using equation (1), above, we have

Thus

In summary, provides upper bound on the magnitude of smallest eigenvalue of R when it is non-singular.

Note: The upper bound determines how slowly the fastest decaying mode approaches zero.

5. Bounds on Decaying Time of G/M/1-type Markov Chain: We now generalize the above derivation of upper bound on smallest eigenvalue of rate matrix to arbitrary G/M/1-type Markov processes. We illustrate the derivation in the case where the degree of matrix polynomial equation is three for notational convenience. It can be readily realized that the derivation essentially generalizes to matrix power series equation. But the notation becomes more complicated. Consider the case where the rate matrix R satisfies a third degree matrix polynomial equation i.e.

In the literature, there are results on bounding the spectral radius of rate matrix of arbitrary G/M/1-type Markov process. But, to the best of our knowledge, there are no results on bounding the smallest eigenvalue of the rate matrix of an arbitrary G/M/1-type Markov process. We specifically consider the case where the rate matrix is non-singular. It can be easily shown that for an arbitrary G/M/1-type Markov process Rank ( R ) = Rank ( Hence, we consider the case where is non-singular. As in the case of QBD process, we have the following interesting Lemma.

Lemma 2:

Proof: It can be easily verified that the following factorization readily holds

Since are the sub-matrices of the generator matrix, it readily follows that , we have that

=

Since the spectral radius of R is strictly less than one, we have that

Hence, (

For a positive recurrent G/M/1-type Markov process, we have that .

Thus, ( is diagonally dominant and hence is non-singular. As in the case of QBD process, it is easy to prove that the inverse of ( ( diagonally dominant matrix with negative diagonal Elements ) is NON-POSITIVE matrix. Thus,

. Hence,

For notational convenience, let us use the following notation: Thus, is a stochastic matrix. Since, rate matrix are non-negative matrices, it follows that is a substochastic matrix. Now, taking the determinant on both sides, we have

Since, the matrix S is sub-stochastic

Hence ( Y ) | . Equivalently, ……...(2).

We readily have that

Thus, we have

Taking determinant on both sides, we have

(

Also |(

Further, since rate matrix R is substochastic, we have that

Thus,

Hence, we have that

Now, using equation (2), we have that

Thus

Note: Using similar reasoning, we can show that in the case of G/M/1-type Markov process sastisfying a matrix polynomial equation, we have

6. Conclusions: In this research paper, motivated by the concept of mixing time of a Markov chain, the concept of decaying time of a skip-free Markov chain is introduced. It is reasoned that the decaying time is related to the smallest as well as largest eigenvalues of the rate matrix. Lower bounds on the magnitude of smallest eigenvalue of QBD process as well as structured G/M/1-type Markov process are derived. It is hoped that these bounds are readily applied for truncating the matrix polynomial/matrix power series arising in the equilibrium analysis of Skip Free Markov chains.

REFERENCES:

(1) [Neu1] M.F.Neuts, Matrix Geometric Solutions in Stochastic Models, Johns Hopkins University Press, Baltimore, 1981

(2) [RKC] G. Rama Murthy, M. Kim and E.J.Coyle, “ Equilibrium Analysis of Skip Free Markov Chains: Nonlinear Matrix Equations , “ Communications in Statistics—Stochastic Models, 7(4), pp. 547-571 , 1991.