OPTIMAL SIGNAL SETS FOR NONGAUSSIAN

DETECTORS

y z

MARK S GOCKENBACH AND ANTHONY J KEARSLEY

Abstract Identifying a maximallyseparated set of signals is imp ortant in the design of mo dems

The notion of optimality is dep endent on the mo del chosen to describ e noise in the measurements

while some analytic results can b e derived under the assumption of Gaussian noise no suchtechniques

are known for choosing signal sets in the nonGaussian case To obtain numerical solutions for non

Gaussian detectors minimax problems are transformed into nonlinear programs resulting in a novel

formulation yielding problems with relatively few variables and many inequality constraints Using

sequential optimal signal sets are obtained for a variety of noise distributions

Key words Optimal Design Inequality Constraints Sequential Quadratic Programming

of the National Institute of Standards and Technology and not sub ject to copyright Contribution

in the United States

y

Department of Mathematics Universityof Michigan Ann Arb or MI

z

Mathematical and Computational Sciences Division National Institute of Standards and Tech

nology Gaithersburg MD

Introduction The transmission of digital information requires signals nite time series

that can b e distinguished from one another in the presence of noise These signals may b e constrained

by b ounds on their energy or amplitude the degree to which they can b e distinguished dep ends on

the distribution of noise

We study the design of optimal signal sets under amplitude constraints and in the presence of

nonGaussian noise We will call a signal set optimal when roughly sp eaking the largest probability

of mistaking any one signal for any other is minimal For this reason as weshowbelow this problem

is naturally formulated as a smo oth and twice continuously dierentiable minimax problem

We shall assume that M signals

s s s

M

are to b e constructed where each signal is to b e a linear combination of K given signals

K

Moreover it is assumed that each is a time series of length N andthat f g is an orthonormal set

k k

under the Euclidean inner pro duct We denote the comp onents of by n n N

k k

and the comp onents of s similarly The unknowns to b e determined are the weights f g dening

m

mk

signals the

K

X

m M s

m

mk k

k

A signal set is often referred to as a constel lation

In the sp ecial case of only two basis functions the signals can b e represented in the plane

by the co ecients In the case of Gaussian noise it turns out

M M

that the problem reduces to maximizing the minimum Euclidean distance b etween anytwo signals

sub ject to constraints on the energy or amplitudes of the signals For this reason heuristic metho ds

have b een used to design go o d signals sets

on a lattice and Typically these heuristics have taken the form of cho osing the points lying

w densely packing them within a xed region of the plane Identifying these latticebased someho

constellations asso ciated with lowaverage energies has b een an active area of research examples and

pictures of these constellations can b e found in and

Although wearenotaware of previous attempts to nd optimal constellations according to the

criterion we describ e b elow related problems have b een investigated The most famous is the sphere

packing problem of communication theory see this requires a constellation which maximizes

the probability of detection under Gaussian noise Mo dern researchinto this question has fo cused on

the case in which the signals are chosen from a large dimensional space in particular an imp ortant

research topic has b een the Strong Simplex Conjecture which deals with the case M K see

We are concerned with the case in which the signals are chosen from a small dimensional space

K or

In the remainder of this pap er we show that this problem can be formulated as a smo oth

problem with relatively few variables but many inequality constraints This

problem is solved using an SQP algorithm

In the following second section we explain the formulation of the optimization problem that

describ es optimal signal sets In Section we describ e the SQP algorithm employed to solve these

problems The noise distributions used in our computations are describ ed in Section and in Section

numerical tests and results are presented We conclude the pap er with observations and comments

ork in Section on future w

Problem formulation As mentioned ab ove we wish to nd a signal set which minimizes

the largest probability of mistaking any one for any other This notion of optimality is therefore based

on hypothesis testing see for example We consider for the moment that one of two signals

s and s is to b e transmitted and that the received signal is denoted by y We assume further

that the transmitted signal is corrupted by indep endent identically distributed iid additive noise

drawn from some xed distribution with probability density function p df p In other words

N

y s m or

m

We assume that the a priori probabilities of s and s b eing transmitted are P and P respec

tively and that there is a cost C asso ciated with detecting signal s when s is actually presenta

m

miss and a cost C asso ciated with detecting s when s is presentafalse alarm It is then easy

f

to show see that the exp ected cost or risk is minimized by detecting s whenever

py js P C

m

py js P C

f

and otherwise detecting s By taking the logarithm of b oth sides and using the fact that the noise

is iid we obtain the following optimal detector Detect s whenever

N

X

p y n s n

N

log

N p y n s n

N

n

where is the threshold determined by the a priori probabilities and the costs of errors and

otherwise detect s

Now assume that s was actually transmitted so that y s and let s s s the

optimal detector then computes

N

X

p n

N

log

s n N p n

N

n

The exp ected value of the n th term is

Z

h i

p

N

K sn log p d

N N

p sn

N

this quan tity is the KullbackLeibler distance b etween the noise density and the noise density shifted

bysn see

Thus if s is actually transmitted the exp ected value of the sum in is

N

X

K sn

N

N

n

one assumes that p is symmetric as we will it is easy to show that if s is transmitted the If

N

ected value of the sum in is the negative of Therefore the probability of detecting exp

the correct signal increases with

From this discussion we see that wewishtocho ose the signals s s s so that

m

N

X

n n s min K s

m m

N



m m



n

is maximized For physical reasons either the average p ower energyL norm or the p eak p ower

amplitudeL norm of the signals must b e constrained In this pap er we are concerned with

amplitude constraints

We are thus faced with a constrained minimax problem Due to the diculty of solving such

problems we rewrite it as a smo oth NLP b ariable t yintro ducing an auxiliary v

min t

t

N

X

n t m m n s st K s

m m

N



n

s n C n N m M

m

t

where C is the b ound on the amplitudes of the signals This problem has MK variables and

M M MN inequality constraints A typical problem would have K M N

giving variables and constraints

Diculties arise when one tries to solve the ab ove NLP with standard algorithms The fact

that there are far few er variables than constraints results in three sp ecic diculties

There are many almost binding constraints at the solution

The linearized constraints are often inconsistent

The b oundary of the feasible region is noticeably nonlinear

The proximity of nearbinding constraints to the solution suggests that correct identication of the

active set b ecomes more dicult near the solution In turn this suggests that the region of rapid

lo cal convergence of the iteration sequence fx g will b e small The lack of consistent linear

k k

inequalities complicates the calculation of the SQP step the solution of the quadratic subproblem

As observed in previous works and others nonlinear feasible regions can result in small ac

ceptable steps if p enalty parameters b ecome small Iterates will follow to o closely a feasible region

b oundary that leads away from optimality All three of these issues were relevanttothesolutionof

the optimal signal set problem presented here

Name Density K s

  

exp s

p

Gaussian





p

jsj expj j jsj

p p p

xp Laplacian e

sech

Hyp erb olic Secant lnsech s

  



s s

Generalized Gaussian exp

   

A

A

Cauchy s r ln



Table

Noise densities and associatedKullbackLeibler distances

Nonlinearly Constrained Optimization Many algorithms have b een develop ed for the

solution of smo oth inequalityconstrained optimization problems Among the most p opular metho ds

is the Sequential Quadratic Programming SQP family of algorithms see for a review of these

metho ds Given an estimate of the solution an SQP algorithm progresses by solving a quadratic

program QP which is dened by a lo cal quadratic mo del of the ob jective function and linearized

constraints The solution to the quadratic program is then used to construct an improved estimate

of the solution Many dierent SQP algorithms can b e constructed byvarying the algorithm for

solving the QP the Lagrange multiplier estimates and the globalization strategy

One version of the SQP algorithm prop osed by Boggs Kearsley and Tolle see for example

app ears to b e well suited for solving this class of NLPs The algorithm employs a combined

trustregion and merit function see pro cedure for globalization Amoderninterior

point metho d called OD Optimal Dimensional subspace metho d see Boggs Domich Rogers

and Witzgall is used to solve the quadratic programming subproblems It app ears that D is

quite compatible with the globalization pro cedure eg the steps pro duced by the OD algorithm

decrease the merit function and do not imp ede convergence

A nonstandard feature of this algorithm is useful b ecause of the need to solve NLPs with very

small and highly nonlinear feasible regions A p erturbation is added to the right hand side of the

system of linearized constraints to guarantee that this linear system is always consistent Similar

Biggs constraint relaxations have app eared in the literature b efore see for instance the pap ers by

owell among others The relaxation pro cedure weemploy is similar to metho ds Tone and P

contained in pap ers mentioned ab ove with minor mo dications and can b e found in When

far from feasibility violated linearized constraints are relaxed enough to guarantee that they form

a consistent system of inequalities This relaxation is obtained by solving a

problem moreover the solution to the linear programming problem can then b e used as a feasible

starting p ointfortheQP Because OD is designed to solve either linear or quadratic programming

problems this twostep pro cess can b e carried out using one algorithm and co de This pro cedure

ensures that no phaseI or infeasible calculations are needed ie no BigM metho d is needed

for the calculation of the SQP descent direction Details of the constraint p erturbation pro cedure

can b e found in

To test the ecacy of the constraint relaxation pro cedure wesolved the problems describ ed

in this pap er twice rst without the relaxation and then with it The advantages of relaxing the

constraints are shown bythe numerical results presented in the following section

Noise distributions The primary purp ose of this pap er is to investigate nonGaussian

noise distributions Following Johnson and Orsak see we selected the ve densities found in

Table including the Gaussian density for comparison These densities are graphed in Figure

while the asso ciated KullbackLeibler distances are found in Figure

These densities are chosen to illustrate dierent p ossibilities For example the KullbackLeibler

distance asso ciated with the Gaussian density is smaller than that of the Laplacian when s is small

yp erb olic secant density leads to a distance function for large s this relationship is reversed The h

that is similar to that of the Gaussian near the origin but close to that of the Laplacian for large

s The KullbackLeibler distance for the Generalized Gaussian density grows very rapidly with s

while that of the Cauchy density grows very slowly

Numerical Tests and Results In this section we summarize the p erformance of the SQP

algorithms on a suite of problems corresp onding to various choices of the noise distribution the basis

signals and the number of signals M For the purp ose of these numerical examples we x the length

p

of the signals at N the amplitude at C and the numb er of basis functions at K Gaussian and Generalized Gaussian densities Laplacian density 0.4 0.8

0.3 0.6

0.2 0.4

0.1 0.2

0 0 −5 0 5 −5 0 5

Hyperbolic Secant density Cauchy density 0.5 0.4

0.4 0.3 0.3 0.2 0.2 0.1 0.1

0 0

−5 0 5 −5 0 5

Fig Noise densities studied

The bases used were

r r

sin nN sin nN

N N

and

r r

sin nN cos nN

N N

with and

We use analytic rst and second derivatives and the leastsquares estimate of the Lagrange

multipliers see for example Gill Murrayand Wright

Problems involving the sinecosine basis are fundamentally more dicult than those involving the

sinesine basis this is b ecause of the rotational symmetries in the solution space and also b ecause of

the geometry of certain signal sets For example when the signal set contains signals optimality

requires that one of the signals lie in the center with signals on a circle around it see Figure

The signal in the center is actually free to move in a small op en set without aecting optimality

Part of the numerical diculty can b e alleviated with a small amount of regularization introduced

to the ob jective function as follows

min t ksk

t

N

X

st K s n s n t m m

m m

N



n

N m M s n C n

m

t Gaussian and Generalized Gaussian Laplacian 5 5

4 4

3 3

2 2

1 1

0 0 −2 −1 0 1 2 −4 −2 0 2 4

Hyperbolic Secant Cauchy 5 2

4 1.5 3 1 2 0.5 1

0 0

−4 −2 0 2 4 −4 −2 0 2 4

Fig Kul lbackLeibler distances associated with the various noise densities

Avalue worked well The problems where this version was employed are denoted byan

asterisk

In Tables and we rep ort the p erformance of the algorithm describ ed in without the

constraint relaxation Likewise Tables and give the p erformance of the algorithm with constraint

relaxation The rst column in the table contains the values for the numberofsignals M the number

of time samples N andthenumber of basis functions K The noise distribution is given in the second

column while the number of variables n and constraints m in the resulting nonlinear program is

found in the third column The number of nonlinear or outer iterations required to nd the solution

is recorded in the fourth column while the fth column gives the numb er of QP iterations required

with the number of phase I iterations in parentheses In the sixth column wegive the value of the

putative global minimum and in the seventh column the number of times this minimum was found

oint In the event that the minimizer in ten tries each starting from a randomly generated starting p

was not found in ten tries we record in parentheses the numb er of tries it to ok to nd it Finally

the last column contains the size of the gradient of the Lagrangian at the computed solution The

algorithm halted when either the norm of the gradient of the Lagrangian b ecame less than or equal

to or the norm of the solution to the quadratic subproblem the SQP step fell b elow

The optimal constellations for a subset of our test problems are shown in Figures and

with M and and with M It is interesting to observe that the symmetric nature

of constellations conjectured to b e present in optimal solutions see is apparent in the current

estimates of the solutions

These problems have features that make them dicult to solvenumerically Not only is the

numberofvariables much smaller than the numb er of constraints but there are many constraints

that are nearly binding at the solution Many algorithms including SQP give rise to rapid lo cal

convergence when iterates enter a neighborho o d of the solution and the correct collection of activesets

has b een identied see the pap er by Robinson for a discussion This rapid lo cal convergence is

esp ecially apparentintheevent that one can provide an accurate approximation to the true Hessian

matrix at every iteration as is the case with our problem Even though the notion of active sets is

less imp ortant to our algorithm b ecause our interior p oint metho d quadratic program solver D

do es not compute active sets the fact that many of the constraints are nearly binding at the solution

MNK density nm outer inner min kr Lk

x

Gaussian e

GeneralizedGaussian e

Hyp erb olicSecant e

Laplacian e

Cauchy e

Gaussian e

GeneralizedGaussian e

Hyp erb olicSecant e

Laplacian e

Cauchy e

Table

Performanceof theunperturbed SQP algorithm on a col lection of constellation problems sine

sine basis

MNK density nm outer inner min kr Lk

x

Gaussian e

GeneralizedGaussian e

Hyp erb olicSecant e

Laplacian e

Cauchy e

Gaussian e

GeneralizedGaussian e

Hyp erb olicSecant e

Laplacian e

Cauchy e

Table

theunperturbed SQP algorithm on a col lection of constellation problems sine Performanceof

cosine basis

has an eect on the size of the neighborho o d around the solution where fast lo cal convergence is

realized

In these numerical tests the constraint relaxation substantially improved the p erformance of the

algorithm This is probably due to the fact that the numb er of inconsistent subproblems encountered

was unusually high It is worth commenting that the additional cost of employing the relaxation

pro cedure is not large

Conclusions In this pap er wehave presented an interesting collection of dicult opti

mization problems and an NLP formulation of them This formulation allows a broad arsenal of

numerical optimization algorithms and mo dern enhancements to b e employed While these prob

lems are not largescale by mo dern computing standards they are nonetheless dicult problems

to solve eciently

Numerical solutions to these problems were lo cated using an SQP metho d with and without the

constraint relaxation pro cedure describ ed in Numerous numerical tests and the summary of

these tests that app ear in Tables and suggest that this constraint relaxation pro cedure

can signicantly improve the p erformance of this SQP metho d in the event that linearizations are

inconsistent whichmay b e the case when there are far more constraints than variables

Because there are so many dierent algorithms and implementations for the solution of the

nonlinear programming problem there is a need to create an accepted collection of test problems

er by Bongartz Conn Gould and Toint Because of the diculties it p oses this see the pap

family of problems is a natural candidate for such a collection

Acknowledgments The authors are grateful to Professor Donald Johnson and his graduate

student DongMei Li at Rice University for manypatient discussions The authors also thank Dr

L C Cowsar from LucentTechnologies Finally the authors greatly appreciated careful reading of

an anonymous referee who suggested numerous b enecial changes to this pap er

MNK density nm outer inner min kr Lk

x

Gaussian e

GeneralizedGaussian e

Hyp erb olicSecant e

Laplacian e

Cauchy e

Gaussian e

GeneralizedGaussian e

Hyp erb olicSecant e

Laplacian e

Cauchy e

Table

Performance of the constr aint perturbed SQP algorithm on a col lection of constellation problems

sinesine basis

MNK density nm outer inner min kr Lk

x

Gaussian e

GeneralizedGaussian e

Hyp erb olicSecant e

Laplacian e

Cauchy e

Gaussian e

GeneralizedGaussian e

Hyp erb olicSecant e

Laplacian e

Cauchy e

Table

aint perturbed SQP algorithm on a col lection of constellation problems Performance of the constr

sinecosine basis

REFERENCES

A V Balakrishnan Acontribution to the spherepacking problem of communication systems

J Math Anal Appl

M C BartholomewBiggs Opsqp and opalqp new implementations of the sequential quad

tratic programming approach to constrained optimisation Technical Rep ort Hateld

Polytechnic Numerical Optimisation Centre

P T Boggs P D Domich J E Rogers and C Witzgall An interior p oint metho d for general

large scale quadratic programming problems Annals of Operations Research

P T Boggs A J KearsleyandJWTolle A practical algorithm for general large scale non

linear optimization problems Technical Rep ort National Institute of Standards and

Technology Gaithersburg MD to app ear SIAM Journal on Optimization

P T Boggs A J Kearsley and J W Tolle A global convergence analysis of an algorithm for

large scale nonlinear programming problems Technical Rep ort National Institute of

Standards and Technology Gaithersburg MD submitted SIAM Journal on

Optimization

P T Boggs and J Tolle Sequential quadratic programming Acta Numerica pages

I Bongartz A R Conn N I M Gould and PLToint Cute Constrained and unconstrained

vironment ACM Trans Math Software testing en

C N Camp opiano and B G Glazer A coherent digital amplitude and phase mo dulation

scheme IRE Trans Commun Syst

T M Cover and B Gopinath Open Problems in Comminication Computation Springer

Verlag New York

G D Forney R G Gallager G R Lang F M Longsta and S U Qureshi Ecient

mo dulation for bandlimited channels IEEE J Sel Areas Commun Sac

G J Foschini R D Gitlin and S B Weinstein Optimization of twodimensional signal

constellations in the prescence of gaussian noise IEE Trans Commun Com 15 15

10 10

5 5

0 0

−5 −5

−10 −10

−15 −15

−15 −10 −5 0 5 10 15 −15 −10 −5 0 5 10 15

Fig Optimal constellation for M Fig Optimal constellation for M

with SineSine basis functions using a Gaus with SineSine basis functions using a Gener

sian Density alized Gaussian Density

15 15

10 10

5 5

0 0

−5 −5

−10 −10

−15 −15

−15 −10 −5 0 5 10 15 −15 −10 −5 0 5 10 15

Fig Optimal constellation for M Fig Optimal constellation for M

with SineCosine basis functions using a with SineSine basis functions using a Cauchy

HyperbolicSecant Density Density

P E Gill W Murray M Saunders and M H Wright Some theoretical prop erties of an

augmented Lagrangian merit function Technical Rep ort Dept Op erations Research

Stanford University Stanford California

P E Gill W Murray and M H Wright Practical Optimization Academic Press Inc

W W Harman Principles of the statistical theory of communication McGrawHill New York

D H Johnson and G C Orsak Performance of optimal nongaussian detectors IEE Trans

Communication

A J Kearsley The use of optimization techniques in the solution of partial dierential equa

tions from science and engineering Technical Rep ort PhD Thesis Rice University

Department of Computational Applied Mathematics

M J D Powell A fast algorithm for nonlinearly constrained optimization calculations In

Proceedings of the Dundee Biennial Conference on Numerical Analysis Berlin

SpringerVerlag

erturb ed KuhnTucker p oints and rates of convergence for a class of nonlinear S Robinson P

programming algorithms Mathematical Programming

K Tone Revisions of constraintapproximations in the successive QP metho d for nonlinear

programming problems Mathematical Programming 20 15

15 10 10

5 5

0 0

−5 −5

−10 −10 −15

−15 −20

−20 −15 −10 −5 0 5 10 15 20 −15 −10 −5 0 5 10 15

Fig Optimal constellation for M Fig Optimal constellation for M

with SineCosine basis functions using a with SineCosine basis functions using a Gen

Cauchy Density eralized Gaussian Density