EURASIP Journal on Applied Signal Processing 2002:5, 525–531 c 2002 Hindawi Publishing Corporation Maximum-Likelihood Sequence Detection of Multiple Antenna Systems over Dispersive Channels viaSphereDecoding Haris Vikalo Information Systems Laboratory, Stanford University, Stanford, CA 94305, USA Email: [email protected] Babak Hassibi Department of Electrical Engineering, California Institute of Technology, Pasadena, CA 91125, USA Email: [email protected] Received 6 May 2001 and in revised form 28 March 2002 Multiple antenna systems are capable of providing high data rate transmissions over wireless channels. When the channels are dispersive, the signal at each receive antenna is a combination of both the current and past symbols sent from all transmit anten- nas corrupted by noise. The optimal receiver is a maximum-likelihood sequence detector and is often considered to be practically infeasible due to high computational complexity (exponential in number of antennas and channel memory). Therefore, in prac- tice, one often settles for a less complex suboptimal receiver structure, typically with an equalizer meant to suppress both the intersymbol and interuser interference, followed by the decoder. We propose a sphere decoding for the sequence detection in multiple antenna communication systems over dispersive channels. The sphere decoding provides the maximum-likelihood esti- mate with computational complexity comparable to the standard space-time decision-feedback equalizing (DFE) algorithms. The performance and complexity of the sphere decoding are compared with the DFE algorithm by means of simulations. Keywords and phrases: sphere decoding, maximum-likelihood, multiple antennas, dispersive channels, computational complexity. 1. INTRODUCTION may be linear (zero-forcing or minimum mean square), Multiple antenna wireless communication systems are capa- or nonlinear decision-feedback equalizer (DFE). DFEs es- ble of providing data transmission at potentially very high sentially perform successive interference cancellation: a soft rates [1]. To secure high reliability of the data transmis- symbol estimate is used to cancel the trailing interference, sion, special attention has to be payed to the design of the upon which the hard decision is made to recover the sym- receiver. When transmitting over noisy dispersive channels, bol. (For the analysis of the performance of DFE algorithm in the received signal at each receive antenna is the combi- a dispersive MIMO environment, see [6].) For high enough nation of the transmitted signals perturbed by noise, in- SNR, DFEs obtain better performance than linear equalizers tersymbol interference (ISI), and by interuser interference while still having much lower complexity than the optimal (IUI). In this case, the optimal receiver structure is the multi- MLSE algorithm. However, the performance of the DFE is channel maximum-likelihood sequence estimation (MLSE). highly inferior compared to the performance of the optimal However, the computational complexity of the traditional MLSE algorithm. maximum-likelihood sequence detector often prohibits its In this paper, we propose an algorithm that yields the practical implementation. (For instance, the Viterbi decoder optimal MLSE performance on dispersive multiple-input is exponential in the length of the channel [2].) One way multiple-output (MIMO) channels with finite impulse re- to alleviate the computational burden is to settle for (sub- sponse (FIR). (We should point out that the wireless commu- optimal) reduced complexity MLSE algorithms by reducing nication systems may or may not employ feedback from the the number of states (see, e.g., [3, 4]). In practice, however, receiver to the transmitter. In this paper, we focus on optimal most often a multichannel (space-time) equalizer is used to detector structures for systems where feedback is unavail- suppress ISI and IUI first; then, a hard decision is made to able and the receiver learns the channel based on the training recover the symbol that has been sent [2, 5, 6]. The equalizer information.) 526 EURASIP Journal on Applied Signal Processing ν(1) We consider the so-called sphere decoding, an algorithm k (1) for solving integer least-squares problems, which, in the s , χ(1) k h(2 1) k communication context, provides the ML estimate of the transmitted data sequence. The algorithm is due to Fincke h(1,2) ν(2) and Pohst [7] and was first proposed in the context of the k s(2) χ(2) closest point searches in lattices (for a review of these, see k k [8] and the references therein). The algorithm was rediscov- ered in [9] in the context of detection in GPS systems. The use of the sphere decoding for lattice codes was first pro- . (N,1) . h . posed in [10],andfurtherinvestigatedin[11, 12]. In [13], . it has been analytically shown that the average complexity ν(N) of the sphere decoding used for ML detection in flat fading k M s( ) χ(N) multiple-antenna systems is polynomial (often sub-cubic) k k for a wide range of signal-to-noise ratios (SNRs). The paper is organized as follows: in Section 2,wede- scribe the FIR MIMO channel model. In Section 3,wepose Figure 1: FIR MIMO channel model. the detection problem, briefly overview heuristics for solving it, and describe the sphere decoding algorithm. Simulation results are presented in Section 4, where it is shown that the matrix form as sphere decoding provides significant improvement (several dBs) over the MIMO DFE. The computational complexity of C ᐄ = H ᏿ ᐂ , the sphere decoding turns out to be comparable to that of the k l k−l + k (3) MIMO DFE, thereby suggesting that it can be implemented l=1 in practice. The paper concludes with Section 5. where ᏿ = s(1) s(2) ··· s(M) 2. FIR MIMO MODEL DESCRIPTION k k k k (4) M We consider a multiple-antenna system with transmit and is the transmit vector, whose entries typically come from a N N× receive antennas. The MIMO channel is modeled as block- QAM constellation, ᐂk ∈ Ꮿ 1 is the additive noise vector fading frequency-selective, where the channel impulse re- defined as sponse is constant for some discrete interval T,afterwhich it changes to another (independent) impulse response that (1) (2) (N) ᐂk = ν ν ··· ν , (5) remains constant for another interval T, and so on. The addi- k k k tive noise is spatially and temporally independent identically H ∈ ᏯN×M l ffi distributed (i.i.d.) circularly-symmetric complex-Gaussian. and l is the th coe cient matrix in the MIMO The MIMO channel model is shown in Figure 1. channel impulse response, The channel is represented by its complex baseband h(1,1) h(1,2) ··· h(1,M) equivalent model. Let the column vector l l l (2,1) (2,2) (2,M) h h ··· h (i,j) = (i,j) (i,j) (i,j) H = l l l . h h h ··· h i,j (1) l (6) 1 2 C( ) . .. . denote the single-input single-output (SISO) channel im- h(N,1) h(N,2) ··· h(N,M) pulse response from the jth transmit to the ith receive an- l l l tenna. For convenience, we shall make the following assump- In other words, the z-transform of the MIMO channel tions on the SISO channels h(i,j): impulse response is given by i,j (1) C( ) = C,1 ≤ i ≤ N,1 ≤ j ≤ M, that is, all SISO −1 −(C−1) H(z) = H1 + H2z + ···+ HCz . (7) channels have impulse responses of the same length, ffi h(i,j) ≤ l ≤ C ≤ i ≤ N (2) the channel coe cients l ,1 ,1 , Define the following vectors: 1 ≤ j ≤ M are i.i.d. Ꮿ(0, 1). ᐄ = ᐄ ᐄ ··· ᐄT C− , 1 2 + 1 The received signal at the ith antenna can then be expressed ᐂ = ᐂ ᐂ ··· ᐂT C− , (8) as 1 2 + 1 ᏿ = ᏿ ᏿ ··· ᏿ . M C 1 2 T i i,j j i χ( ) = h( )s( ) + ν( ), (2) k l k−l k ᐂ ∈ ᏯN(T+C−1) j=1 l=1 (Note that the random vector has unit vari- ∗ ance complex Gaussian i.i.d. entries, E[ᐂᐂ ] = IN(T+C−1).) for k = 1, 2,...,T + C − 1. Equation (2)canbewrittenina Then from (3) we can write the input-output relation for the MIMO MLSD via Sphere Decoding 527 FIR MIMO channel in the matrix form as ᐂ ᐄ = Ᏼ᏿ ᐂ, ᏿ ᐄ + (9) Ᏼ where Ᏼ ∈ ᏯN(T+C−1)×MT is constructed as Figure 2: Matrix equivalent channel model. H1 H H 2 1 as . . .. x = Hs + v, (14) HC ··· H 1 Ᏼ = .. .. . (10) . where the signal vectors s are typically obtained upon mod- H ··· H L Ᏸ2MT C 1 ulation of the input bits onto an -PAM constellation L , .. . H H L − L − L − L − 2MT C C−1 2MT 1 3 3 1 ᏰL = − , − ,..., , . (15) HC 2 2 2 2 Model (9)isillustratedinFigure 2. We assume that symbol (This particular structure of vector s stems from the assump- ᏿ L × L bursts are uncorrelated (which is an appropriate assumption tion that entries of in (9) are points in QAM constel- L when modeling, for instance, packet transmission in TDMA lation.) Notice that we assumed that is even. (In practice, L systems). is commonly a power of 2, giving rise to 2-PAM, 4-PAM, It will be convenient to define the signal-to-noise ratio ρ 8-PAM, etc., constellations.) Ᏸ2MT for the system in (9), Finally, notice that L is a finite lattice carved from an infinite one, ᐆ2MT. EᏴ᏿2 ρ = 2 Eᐂ2 3. PROBLEM STATEMENT 2 E tr Ᏼ᏿᏿∗Ᏼ∗ With the notation introduced in Section 2, due to the Gaus- = (11) E tr ᐂᐂ∗ sian assumption on the additive noise, we can express the MLSE problem as the optimization problem E tr ᏿᏿∗Ᏼ∗Ᏼ = . N(T + C − 1) min x − Hs2, (16) 2MT 2MT s∈ᏰL ⊂ᐆ Assuming that the entries in ᏿ are coming from an L×L QAM L where the minimization is over all points in the constellation constellation (where is assumed to be even), and that the 2MT ᏰL .