IEEE Information Theory Society Newsletter Vol. 53, No.4, December 2003 Editor: Lance C. Pérez ISSN 1059-2362 The Shannon Lecture Hidden Markov Models and the Baum-Welch Algorithm Lloyd R. Welch Content of This Talk what the ‘running variable’ is. The lectures of previous Shannon Lecturers fall into several Of particular use will be the concept of conditional probabil- categories such as introducing new areas of research, resusci- ity and recursive factorization. The recursive factorization tating areas of research, surveying areas identified with the idea says that the joint probability of a collection of events can lecturer, or reminiscing on the career of the lecturer. In this be expressed as a product of conditional probabilities, where talk I decided to restrict the subject to the Baum-Welch “algo- each is the probability of an event conditioned on all previous rithm” and some of the ideas that led to its development. events. For example, let A, B, and C be three events. Then I am sure that most of you are familiar with Markov chains Pr(A ∩ B ∩ C) = Pr(A)Pr(B | A)Pr(C | A ∩ B) and Markov processes. They are natural models for various communication channels in which channel conditions change Using the bracket notation, we can display the recursive fac- with time. In many cases it is not the state sequence of the torization of the joint probability distribution of a sequence of model which is observed but the effects of the process on a discrete random variables: signal. That is, the states are not observable but some func- tions, possibly random, of the states are observed. In some X( ) N= N = (X( ) = )· Pr [ k ]0 [xk]0 Pr 0 x0 cases it is easy to assign the values of the parameters to model N a channel. All that remains is to determine what probabilities X( )= | X( ) n−1 = n−1 Pr n xn [ k ]0 [xk]0 are desired and derive the necessary algorithms to compute n=0 them. Markov Chains and Hidden Markov In other cases, the choice of parameter values is only an esti- Chains mate and it is desired to find the “best” values. The usual cri- terion is maximum likelihood. That is: find the values of We will treat only Markov Chains which have finite state parameters which maximizes the probability of the observed spaces. The theory is more general, but to cover the more gen- data. This is the problem that the Baum-Welch computation eral case will only obscure the basic ideas. addresses. Let S be a finite set, the set of states. Let the number of ele- Preliminaries ments in S be M. It will be convenient to identify the ele- ments of S with the integers from 1 to M. Let N be the set of non-negative integers. Let’s introduce some useful notation to replace the usual n-tuple notations: Let {S(t) : t ∈ N } be a sequence of random variables with Pr(S(t) ∈ S) = 1 for all t ∈ N . That is, the values of S(t) are j confined to S. ≡ ( , + ,... , ) [ak]k=i ai ai 1 aj ( ) j ≡ ( ( ), ( + ),... , ( )) [a k ]k=i a i a i 1 a j Applying the above factorization to the joint distribution of the first N + 1 random variables gives: The ‘k =’ will be dropped from the subscript when it is clear continued on page 10 2 From the Editor Lance C. Pérez This issue of the IEEE Information Theory Society Newsletter high- Electronic submission, especially in lights Lloyd Welch's article “Hidden Markov Models and the ascii and Word formats, is encouraged. Baum-Welch Algorithm” based on his Shannon Lecture at the 2003 International Symposium on Information Theory. Once again sev- I may be reached at the following eral members of the IT community have received major awards address: including an Emmy! Please see the announcements on pages 4 and 5 of this issue for details. Lance C. Pérez Department of Electrical Engineering Please help make the Newsletter as interesting and informative as 209N Walter Scott Engineering Center possible by offering suggestions and contributing news. The dead- University of Nebraska-Lincoln lines for the 2004 issues of the newsletter are as follows: Lincoln, NE 68588-0511 Phone: (402)472-6258 Issue Deadline Fax: (402)472-4732 March 2004 January 15, 2004 Email: [email protected] June 2004 April 15, 2004 September 2004 July 15, 2004 December 2004 October 15, 2004 Sincerely, Lance C. Pérez IEEE Table of Contents Information Theory Society Newsletter Hidden Markov Models and the Baum-Welch Algorithm . .1 From the Editor . .2 IEEE Information Theory Society Newsletter (USPS 360-350) is published quarterly by the President’s Column . .3 Information Theory Society of the Institute of Call for Nominations for the 2004 IEEE Information Theory Society Paper Award . .4 Electrical and Electronics Engineers, Inc. Call for Nominations for the 2004 Joint Information Theory /Communications Society Headquarters: 3 Park Avenue, 17th Floor, Paper Award . .4 New York, NY 10016-5997. 2003 Paper Award Winners Announced . .4 Cost is $1.00 per member per year (included Emmy Award For Kees Immink . .5 in Society fee) for each member of the Information Theory Society. Printed in the Historians Column . .6 U.S.A. Periodicals postage paid at New York, NY and at additional mailing offices. Information Theorist Robert Gallager of MIT and Engineer-Entrepreneur Bob Metcalfe Selected For Prestigious Marconi International Fellowships . .7 Postmaster: Send address changes to IEEE Information Theory Society Newsletter, Golumb’s Puzzle Solutions . .8 IEEE, 445 Hoes Lane, Piscataway, NJ 08854. Gerard J. Foschini Named Bell Labs Fellows . .8 © 2003 IEEE. Information contained in this Golumb’s Puzzle . .9 newsletter may be copied without permis- sion provided that the copies are not made 2003 Fall Workshop on Information Theory & Communications . .14 or distributed for direct commercial advan- tage, and the title of the publication and its Board of Governors Meeting . .15 date appear. Conference Calendar . .24 IEEE Information Theory Society Newsletter December 2003 3 President’s Column Han Vinck In 2003, an IEEE research project measured mum price. Tutorials are well appreciat- membership satisfaction with each society ed and we will continue to have tutorials and the IEEE. A total of 478 IT members in future symposia. responded to the questionnaire. I want to thank the participants for their valuable Transactions: (1) Many respondents com- input. The figure below illustrates the main plain about the time between submission result for the question about IT Society and final publication in the transactions. (ITS) membership satisfaction and IEEE This problem was discussed in the June membership satisfaction. BoG meeting and in the September 2003 President’s column. We are currently test- The general conclusions are: (1) Most of the ing a web based editorial tool developed members are very to highly satisfied with by Publications Editor Kevin Quirk. The ITS membership; (2) The IT transactions are software is intended to simplify the elec- rated as extremely useful by a very high tronic submission, reviewing and tracking percentage of our members and, together of submitted manuscripts; (2) It has also with the newsletter, are the main reason for been suggested that the ITS consider joining the ITS; (3) Chapter activities and the introduction of a new "fast-track" publicity need improvement; and (4) The letters publication similar to the IEEE symposium (ISIT) and workshop (ITW) are IT Society President Han Vinck in front of Communications Letters for timely dis- considered to be of great interest by about Allerton House, location of the 41st Annual semination of research. This idea is a logi- 50% of our members. These outcomes are Allerton Conference. cal consequence of the previous complaint very stimulating for the volunteers work- and could be a point of attention for the ing for you in all kind of positions. Participants in the question- future; (3) The Transactions should be more reader friendly and naire also indicated how we could improve our service to the more survey and tutorial papers are requested. From the responses members. I will summarize and briefly comment on the reactions. I conclude that we have to improve the presentation of fundamen- tal results in order to achieve a broader audience. If we want to Conferences: Members suggested longer summaries (up to 4 attract young researchers to our work and promote information the- pages), which requires a CD/DVD version of the proceedings, ory, we need to present the work in an understandable way, which and the participation fee is considered to be high. Conference fees even includes small worked out examples. This is also of interest to will remain a discussion item for the BoG. We try to minimize the the authors! All together, our journals are highly ranked within and costs of our symposium, but we also want to include a banquet outside the IEEE. The Editorial Board members under the leader- for all, an award luncheon, a get-together party and nice coffee ship of Editor in Chief Paul Siegel deserve great respect for their breaks. In addition, for 30% of the participants we want a reduced dedication as Information Theory Society volunteers. student participation fee and about 50 participants from low- income countries need support. Last, but not least, perfect organ- Volunteers are the basis of our society. They keep the operational ization is expected. Obviously, these requirements lead to a mini- costs very low and create professional friendships. We need your input to maintain our high technical standards and the quality of our organized events. A point of concern is the chapter activity. The chapters act on a local level and are of direct importance to our mem- bers and students.