SOLVING HIDDEN NON-MARKOVIAN MODELS: HOW TO COMPUTE CONDITIONAL STATE CHANGE PROBABILITIES Claudia Krull(a), Graham Horton(b) (a)(b)University of Magdeburg, Department of Simulation and Graphics (a)[email protected], (b)[email protected] ABSTRACT of real life systems. These HnMM can in theory analyze Current production systems produce a wealth of the recorded protocols of not observed systems and information in the form of protocols, which is hardly determine the protocol probability or the most likely exploited, often due to a lack of appropriate analysis generating behavior. methods. Hidden non-Markovian Models attempt to Solving HnMM is still an issue of current research. relieve this problem. They enable the analysis of not HnMM can be analyzed using Proxel-based simulation observable systems only based on recorded output. The (Lazarova-Molnar 2005, Wickborn et al. 2006, Krull paper implements and tests solution algorithms based and Horton 2008), which is however quite expensive, on the original Hidden Markov Model algorithms. The because it computes the complete reachable model state problem of computing the conditional state change space and consequently all possible generating paths. probabilities at specific points in time is solved. The An alternative solution method was proposed in (Krull implemented algorithms compute the probability of a and Horton 2009), where the theory for HnMM was given output sequence and the most probable generating described as well as a theoretical transfer of the original path for Markov regenerative models. This enables the HMM solution algorithms to the new paradigm. analysis of protocols of not observable systems and the This paper shows an implementation of the solution of problems that cannot be solved using adapted HMM algorithms. It also solves the problem of existing methods. This opens up new application areas, how to compute the conditional state change including a different approach to mining the wealth of probabilities at a given point in time in continuous time. data already collected today with huge monetary effort. This is not trivial, since in theory the probability of a state change via a continuous distribution function at a Keywords: hidden non-Markovian models, Hidden specific point in time is 0. Furthermore the performance Markov Models, discrete stochastic models. of the implemented algorithms is tested under growing model size and growing protocol length. A comparison 1. INTRODUCTION with the current Proxel-based solution algorithm will Today machines and electronic components are getting show the differences of the two approaches. The more and more complex. As a result of that growing adapted HMM algorithms are much faster and enable complexity they can produce a wealth of information on the analysis of longer sequences and larger models. the system behavior in the form of protocols. These However, the algorithms only compute the cumulative protocols are most of the time discarded or simply state probability or the one most likely generating path. archived and not analyzed properly, often due to a lack Therefore both solution methods have different of appropriate tools. Another problem is, that often the application areas and should be developed further to be protocol entries are ambiguous. An error for example applicable to real world problems. could have been produced by several internal causes and each cause could also result in different errors. If 1.1. Possible Application Areas of HnMM that is the case, finding the exact or even a probable Application areas encompass all engineering problems system behavior that produced the given protocol is where the true system or machine state is not detectable almost impossible using existing methods. or has not been stored and only protocols containing The recently developed paradigm of Hidden non- ambiguous records are available. A possible application Markovian Models (HnMM) strives to relieve this area outside of engineering could be the analysis of the problem. HnMM are an extension of Hidden Markov symptoms of a patient, which can be interpreted as the Models to more general discrete stochastic hidden output of the unobservable true state of his or her models. HMM can analyze hidden behavior based only disease. on recorded system output. However, the hidden model Today in some parts of the industry a huge effort is is a discrete-time Markov chain and therefore severely invested in collecting data on existing production lines limited. The extension to discrete stochastic hidden by installing internal sensors and logging the observed models enables more realistic models and the analysis processes. The resulting amount of data is often huge and too often not consistent, since the calibration and accurately represent memoryless processes and are coordination of different sensors can be very difficult. therefore not very realistic for most applications. Consequently the analysis of the data can be tedious. Current extensions of HMM focus on increasing the Often the effort that would need to be invested to gain speech recognition capability (Fink 2008). Most of reliable sensor equipment and data analysis is larger them change the output process from discrete to than the expected benefit of such a step. continuous. The few attempts to generalize the hidden A concrete industry application of HnMM could process of an HMM were abandoned due to the minimal therefore be the following. A small production line is increase in speech recognition capability, compared to being observed over a limited period of time through the large increase in computational complexity of the external sensors such as cameras or photoelectric solution algorithms. barriers. These sensors can only detect the visible part of the production process and might not be able to 2.2. Hidden non-Markovian Models observe the individual machines’ or products’ status or The extension of HMM to more general models leads to a hidden buffer content. The analysis of these logs can so-called Hidden non-Markovian Models (HnMM). help determine the working status of the machines and These have been formalized in (Krull and Horton 2009) their current performance indices, helping the owner of and the difference to HMM is twofold. HnMM use the the production line in his decisions regarding the state space of a discrete stochastic model (DSM) as modernization or replacement of machines. Hidden hidden model, enabling non-Markovian state models are necessary in this context, since only a transitions. Furthermore, they associate the symbol portion of the production process can be observed. emissions with the state changes, since these are most However, HMM are not sufficient, since a production often the objects of interest in a DSM. process is most likely time dependent and includes The Proxel-based simulation algorithm was several non-Markovian transitions. Only the extension proposed as HnMM solution method even before to Hidden non-Markovian Models can provide a time formalization (Wickborn et al. 2006, Krull and Horton dependent model for the hidden process. 2008). The Proxel algorithm is a state space-based The application of the proposed method and simulation method that discovers all possible system HnMM analysis in this example could shift and developments in discrete time steps (Horton 2002, considerably reduce the necessary investment for sensor Lazarova-Molnar 2005). These possible development equipment and resulting effort for data collection, paths can correspond to generating paths of HnMM enabling also smaller firms to conduct such studies. output sequences. The solution of HnMM via Proxels is however quite expensive. The method explores all 2. STATE OF THE ART possible generating paths and has therefore a large memory and runtime requirement and is limited to very 2.1. Hidden Markov Models small models. Hidden Markov Models (HMM) (Rabiner 1989, Fink The goal of this paper is to implement the adapted 2008) are a well known paradigm in the realm of speech HMM algorithms presented by Krull and Horton (2009) and pattern recognition. They are used to determine the that promise to be more efficient. One major challenge hidden meaning of an observed output in the form of is the computation of conditional state change recorded speech or scanned pictures or letters. HMM probabilities at given points in time in continuous time. consist of a discrete-time Markov chain (DTMC) This would enable the analysis of larger and therefore (Bolch et. al 2006) as hidden model and a second more realistic models in order to increase the practical stochastic process emitting symbols with given applicability of HnMM. probabilities based on the current system state. Given a sequence of output symbols, one can solve the 3. ADAPTED HMM ALGORITHMS following three tasks: 3.1. Existing HnMM Theory • Evaluation: Determine the probability of that This section gives a brief definition of HnMM and their sequence for a given model. components. A more detailed formalization can be • Decoding: Determine the most likely found in Krull and Horton (2009). A Hidden non- generating state sequence for a given model. Markovian Model contains the state space of a discrete • Training: Train an initialized model to produce stochastic system as hidden model. The symbol outputs that sequence most likely. are associated to the transitions. Algorithms exist to solve all of these problems. ,,,,, (1) The evaluation and decoding problem are solved through iterative algorithms. Training is by far the most An HnMM is represented by a 6-tuple as seen in complex task of the three and is usually solved through Equation (1). The first three elements are the set of an optimization-like method. discrete system states S, the set of state changes C, the The applicability of HMM to our area of interest is set of output symbols V. The transition behavior is limited due to DTMCs as hidden model. These can only described by matrix A, which contains the instantaneous rate functions (IRF) (Horton 2002) of the state of the symbol emission there are multiple state changes transitions.