Topics in Statistical Signal Processing for Estimation and Detection in Wireless Communication Systems by Ido Nevat B.Sc. (Electrical Engineering), Technion - Institute of Technology, Israel, 1998. A thesis submitted in partial fulfillment for the degree of Doctor of Philosophy in the Faculty of Engineering School of Electrical Engineering and Telecommunications The University of New South Wales December 2009 Originality Statement I hereby declare that this submission is my own work and to the best of my knowledge it contains no materials previously published or written by another person, or substantial proportions of material which have been accepted for the award of any other degree or diploma at UNSW or any other educational institution, except where due acknowledgment is made in the thesis. Any contribution made to the research by others, with whom I have worked at UNSW or elsewhere, is explicitly acknowledged in the thesis. I also declare that the intellectual content of this thesis is the product of my own work, except to the extent that assistance from others in the project’s design and conception or in style, presentation and linguistic expression is acknowledged. Signature: Date: iii Copyright Statement I hereby grant the University of New South Wales or its agents the right to archive and to make available my thesis or dissertation in whole or part in the University libraries in all forms of media, now or here after known, subject to the provisions of the Copyright Act 1968. I retain all proprietary rights, such as patent rights. I also retain the right to use in future works (such as articles or books) all or part of this thesis or dissertation. I also authorise University Microfilms to use the 350 word abstract of my thesis in Dissertation Abstract International. I have either used no substantial portions of copyright material in my thesis or I have obtained permission to use copyright material; where permission has not been granted I have applied/will apply for a partial restriction of the digital copy of my thesis or dissertation. Signature: Date: Authenticity Statement I certify that the Library deposit digital copy is a direct equivalent of the final officially approved version of my thesis. No emendation of content has occurred and if there are any minor variations in formatting, they are the result of the conversion to digital format. Signature: Date: iv “What is a scientist after all? It is a curious man looking through a keyhole, the keyhole of nature, trying to know what’s going on.” Jacques Yves Cousteau THE UNIVERSITY OF NEW SOUTH WALES Abstract Faculty of Engineering School of Electrical Engineering and Telecommunications Doctor of Philosophy by Ido Nevat During the last decade there has been a steady increase in the demand for incorporation of high data rate and strong reliability within wireless communication applications. Among the different solutions that have been proposed to cope with this new demand, the utilization of multiple antennas arises as one of the best candidates due to the fact that it provides both an increase in reliability and also in information transmission rate. A Multiple Input Multi- ple Output (MIMO) structure usually assumes a frequency non-selective characteristic at each channel. However, when the transmission rate is high, the whole channel can become frequency selective. Therefore, the use of Orthogonal Frequency Division Multiplexing (OFDM) that transforms a frequency selective channel into a large set of individual frequency non-selective narrowband channels, is well suited to be used in conjunction with MIMO systems. A MIMO system employing OFDM, denoted MIMO-OFDM, is able to achieve high spectral efficiency. However, the adoption of multiple antenna elements at the transmitter for spatial transmission results in a superposition of multiple transmitted signals at the receiver, weighted by their corresponding multipath channels. This in turn results in difficulties with reception, and imposes a real challenge on how to design a practical system that can offer a true spectral efficiency improvement. In addition, as wireless networks continue to expend in geographical size, the distance between the source and the destination precludes direct communication between them. In such scenarios, a repeater is placed between the source and the destination to achieve end-to-end communica- tion. New advances in electronics and semiconductor technologies have enabled and made relay based systems feasible. As a result, these systems have become a hot research topic in the wireless research community in recent years. Potential application areas of cooperation diversity are the next generation cellular networks, mobile wireless ad-hoc networks, and mesh networks for wireless broadband access. Besides increasing the network coverage, relays can provide ad- ditional diversity to combat the effects of the wireless fading channel. This thesis is concerned with methods to facilitate the use of MIMO, OFDM and relay based systems. In the first part of this thesis, we concentrate on low complexity algorithms for detection of symbols in MIMO systems, with various degrees of quality of channel state information. First, we design algorithms for the case that perfect Channel State Information (CSI) is available at the receiver. Next, we design algorithms for the detection of non-uniform symbols constellations where only partial CSI is given at the receiver. These will be based on non-convex and stochastic optimisa- tion techniques. The second part of this thesis addresses primary issues in OFDM systems. We first concentrate on a design of an OFDM receiver. First we design an iterative receiver for OFDM systems which performs detection, decoding and channel tracking that aims at minimising the error propaga- tion effect due to erroneous detection of data symbols. Next we focus our attention to channel estimation in OFDM systems where the number of channel taps and the power delay profile are both unknown a priori. Using Trans Dimensional Markov Chain Monte Carlo (TDMCMC) methodology we design algorithms to perform joint model order selection and channel estimation. The third part of this thesis is dedicated to detection of data symbols in relay systems with non-linear relay functions and where only partial CSI is available at the receiver. In order to design the optimal data detector, the likelihood function needs to be evaluated at the receiver. Since the likelihood function cannot be obtained analytically or not even in a closed form in this case, we shall utilse a “Likelihood Free” inference methodology. This will be based on the Approximate Bayesian Computation (ABC) theory to enable the design of novel data sequence detectors. Acknowledgements During the course of this thesis I have met and interacted with many interesting people, who influenced the path of my research. I would first like to thank my academic supervisor Dr. Jinhong Yuan, for supporting me over the years, and for giving me the freedom to explore my ideas. His constant friendly attitude, encouragement, technical insights and constructive criticism have been invaluable. I would also like to thank him for his financial support that enabled me to give my research work full atten- tion and travel to international conferences. I have been very fortunate to work with Dr. Gareth Peters of School of Mathematics at Uni- versity of NSW. His patience and willingness to share his vast knowledge is inestimable. I will cherish our fruitful discussions and his helpful comments, both on the whiteboard and during our numerous coffee breaks. Many thanks go to Dr. Miquel Payar´oof CTTC, Barcelona, who helped me while I was very confused in the beginning of my studies. I am most grateful to Dr. Ami Wiesel and Dr. Yonian Eldar of Electrical Engineering at the Technion for helping me get a better understanding of Bayesian inference and optimisation tech- niques. Thank you also goes to Dr. Scott Sisson and Dr. Yanan Fan of School of Mathematics at University of NSW for sharing their knowledge in Bayesian Statistics. I would also like to thank all my friends and colleagues at the Wireless Communication Lab at UNSW with whom I have had the pleasure of working over the years; thank you goes to Imitiaz, Tom, Adeel, Marwan, Anisul, Jonas, David, Giovanni, Tuyen and Nam Tram. Thank you also goes to UNSW staff, especially to Joseph Yiu who was willing to land a hand with any technical matter and to Gordon Petzer and May Park for taking care of all adminis- trative issues. Many thanks also go to the people of ACoRN, especially Dr. Lars Rasmussen and Christine Thursby for arranging academic activities and making sure we have sufficient funding. I would like to thank my family for their unwavering support and love throughout my life; without them, none of this would have been possible. Finally I would like to thank my partner Dr. Karin Avnit, for too many reasons to list here. viii Dedicated to my Parents “And since you know you cannot see yourself, so well as by reflection, I, your glass, will modestly discover to yourself, that of yourself which you yet know not of.” William Shakespeare ix Contents Originality Statement iii Copyright Statement iv Authenticity Statement iv Abstract vi Acknowledgements viii List of Figures xix List of Tables xxi List of Algorithms xxiii Acronyms xxiv Notations and Symbols xxx 1 Introduction 1 1.1 Motivation . 1 1.2 Outline of the dissertation . 4 1.3 Research contributions . 6 2 Bayesian Inference and Analysis 9 2.1 Introduction . 9 2.2 Background . 9 2.3 Bayesian Inference . 10 2.3.1 Prior Distributions . 11 2.3.2 Point Estimates . 13 2.3.3 Interval Estimation . 16 2.4 Bayesian Model Selection . 16 xi 2.4.1 Introduction . 16 2.5 Bayesian Estimation Under Unknown Model Order . 18 2.5.1 Bayesian Model Averaging .