Interval Methods for Global Optimization by Belaid Moa B.Eng., Ecole Hassania des Travaux Publics, Morocco, 1998 M.Eng., Ecole Nationale Superieure´ d’Electrotechnique, d’Electronique, d’Informatique, d’Hydraulique et des Tel´ ecommunications,` France, 2000

A Dissertation Submitted in Partial Fulfillment of the Requirements for the Degree of Doctor of Philosophy

in the Department of Computer Science

c Belaid Moa, 2007 University of Victoria All rights reserved. This dissertation may not be reproduced in whole or in part by photocopy or other means, without the permission of the author. Interval Methods for Global Optimization by Belaid Moa B.Eng., Ecole Hassania des Travaux Publics, Morocco, 1998 M.Eng., Ecole Nationale Superieure´ d’Electrotechnique, d’Electronique, d’Informatique, d’Hydraulique et des Tel´ ecommunications,` France, 2000

Supervisory Committee

Dr. M.H. van Emden, Supervisor (Department of Computer Science)

Dr. W.W. Wadge, Co-supervisor (Department of Computer Science)

Dr. J. Ellis, Department Member (Department of Computer Science)

Dr. F. Gebali, Outside Member (Department of Electrical and Computer Engineering)

Dr. M. Ibnkahla, External Examiner (Department of Electrical Engineering, Queen’s University) Supervisory Committee

Dr. M.H. van Emden, Supervisor (Department of Computer Science)

Dr. W.W. Wadge, Co-supervisor (Department of Computer Science)

Dr. J. Ellis, Department Member (Department of Computer Science)

Dr. F. Gebali, Outside Member (Department of Electrical and Computer Engineering)

Dr. M. Ibnkahla, External Examiner (Department of Electrical Engineering, Queen’s University)

ABSTRACT

We propose interval arithmetic and interval constraint algorithms for global optimization. Both of these compute lower and upper bounds of a function over a box, and return a lower and an upper bound for the global minimum. In interval arithmetic methods, the bounds are computed using interval arithmetic evaluations. Interval constraint methods instead use domain reduction operators and consistency algorithms. The usual interval arithmetic algorithms for global optimization suffer from at least one of the following drawbacks:

Mixing the fathoming problem, in which we ask for the global minimum only, with the localization problem, in which we ask for the set of points at which the global

minimum occurs. Not handling the inner and outer approximations for -minimizer, which is the set of iv

points at which the objective function is within of the global minimum.

Nothing is said about the quality for their results in actual computation. The prop- erties of the algorithms are stated only in the limit for infinite running time, infinite memory, and infinite precision of the floating-point number system.

To handle these drawbacks, we propose interval arithmetic algorithms for fathoming prob- lems and for localization problems. For these algorithms we state properties that can be verified in actual executions of the algorithms. Moreover, the algorithms proposed return the best results that can be computed with given expressions for the objective function and the conditions, and a given hardware. Interval constraint methods combine interval arithmetic and constraint processing tech- niques, namely consistency algorithms, to obtain tighter bounds for the objective function over a box. The basic building block of interval constraint methods is the generic propa- gation algorithm. This explains our efforts to improve the generic propagation algorithm as much as possible. All our algorithms, namely dual, clustered, deterministic, and selec- tive propagation algorithms, are developed as an attempt to improve the efficiency of the generic propagation algorithm. The relational box-consistency algorithm is another key algorithm in interval constraints. This algorithm keeps squashing the left and right bounds of the intervals of the variables until no further narrowing is possible. A drawback of this way of squashing is that as we proceed further, the process of squashing becomes slow. Another drawback is that, for some cases, the actual narrowing occurs late. To address these problems, we propose the following algorithms:

1. Dynamic Box-Consistency algorithm: instead of pruning the left and then the right bound of each domain, we alternate the pruning between all the domains.

2. Adaptive Box-Consistency algorithm: the idea behind this algorithm is to get rid of the boxes as soon as possible: start with small boxes and extend them or shrink them depending on the pruning outcome. This adaptive behavior makes this algorithm v

very suitable for quick squashing.

Since the efficiency of interval constraint optimization methods depends heavily on the sharpness of the upper bound for the global minimum, we must make some effort to find the appropriate point or box to use for computing the upper bound, and not to randomly pick one as is commonly done. So, we introduce interval constraints with exploration. These methods use non-interval methods as an exploratory step in solving a global opti- mization problem. The results of the exploration are then used to guide interval constraint algorithms, and thus improve their efficiency. Table of Contents

Supervisory Committee ii

ABSTRACT iii

Table of Contents vi

List of Tables xi

List of Figures xii

List of Abbreviations xvi

Notations xvii

Glossary xix

Acknowledgement xxii

Dedication xxiii

1 Introduction 1 1.1 Motivation ...... 1 1.2 The problem statement ...... 2 1.3 Methods for solving global optimization problems ...... 4 1.4 Layout of the dissertation ...... 6 Table of Contents vii

2 Interval Arithmetic 10 2.1 From floating-point numbers to interval arithmetic ...... 10 2.2 Advantages of interval arithmetic ...... 11 2.3 Basics of interval arithmetic ...... 12 2.3.1 Intervals ...... 12 2.3.2 Interval extensions and Inclusion functions ...... 14 2.4 Solving inequalities via interval arithmetic ...... 16 2.5 Functional box consistency ...... 18

3 Unconstrained global optimization via interval arithmetic 21 3.1 Motivation ...... 21 3.2 The fathoming problem ...... 24 3.2.1 The first fathoming problem ...... 25 3.2.2 The second fathoming problem ...... 27 3.3 The localization problem ...... 34 3.3.1 The outer approximation ...... 34 3.3.2 The inner approximation ...... 37

4 Constrained global optimization via interval arithmetic 40 4.1 Overview ...... 41 4.2 The feasibility problem ...... 43 4.2.1 The outer approximation ...... 44 4.2.2 The inner approximation ...... 45 4.3 The fathoming problem ...... 49 4.4 The localization problem ...... 53

5 Interval constraints 59 5.1 Constraint satisfaction problems ...... 59 5.1.1 Constraints ...... 60 Table of Contents viii

5.1.2 Constraint propagation ...... 62 5.2 Interval constraints and interval constraint solver ...... 63

6 Dual, clustered, deterministic, and selective propagation algorithms 70 6.1 Introduction ...... 71 6.2 Relations and Recursive Functions on Power Sets ...... 72 6.3 Dual Propagation Algorithms ...... 74 6.4 Clustered Propagation Algorithms ...... 77 6.5 Deterministic propagation algorithms ...... 79 6.6 Selective propagation algorithms ...... 80 6.7 Implementation ...... 80

7 Dynamic and adaptive box-consistency algorithms 85 7.1 Box consistency ...... 85 7.2 Dynamic box consistency ...... 86 7.3 Adaptive box consistency ...... 88 7.4 Other improvements ...... 91

8 Global optimization via interval constraints 95

8.1 Overview ...... 95

8.1.1 Computing a lower bound for Ñ
Ò ...... 96

8.1.2 Computing an upper bound for Ñ
Ò ...... 96 8.1.3 Branch-and-bound in interval constraints ...... 97 8.2 Exploration and interval constraints for global optimization ...... 98 8.2.1 The fundamental rule of P-NP ...... 98 8.2.2 Motivation of exploration ...... 98

8.2.3 Interval constraints with exploration ...... 99

8.2.4 Effect of exploration on the upper bound of Ñ
Ò ...... 101

8.2.5 Effect of exploration on the lower bound of Ñ
Ò ...... 101 Table of Contents ix

8.2.6 Effect of exploration on splitting the search space ...... 102 8.2.7 Propagating or squashing? ...... 104 8.3 Interval constraints as a verification tool ...... 107 8.4 Interval constraint solver with exploration ...... 108 8.4.1 Effect of exploration on splitting and propagation ...... 109

8.4.2 Effect of exploration on squashing algorithms ...... 109

8.4.3 Does a box contain a solution to  ? ...... 110

8.4.3.1  involves inequalities only ...... 111

8.4.3.2  involves equalities also ...... 114

9 Experimental results 117 9.1 Description of BelSyste ...... 117 9.1.1 BelSyste architecture ...... 117 9.1.2 BelSyste components ...... 119 9.2 Experimental results ...... 121 9.2.1 An application using squashing and propagation ...... 121 9.2.2 Box consistency ...... 123 9.2.3 Box satisfaction ...... 125 9.2.4 Box unsatisfaction ...... 126 9.2.5 Largest box in a convex set ...... 126 9.2.6 Booth function ...... 131 9.2.7 Rosenbrock banana function ...... 133 9.2.8 Six-hump camel function ...... 134 9.2.9 Discrete examples ...... 137 9.2.10 An inequality on integers ...... 137 9.2.11 Integral points on elliptic curves ...... 138 9.2.11.1 Fermat’s last theorem ...... 139 9.2.11.2 Taxicab number ...... 139 Table of Contents x

9.2.12 BelSyste as a verification tool ...... 140

10 Conclusion 143 10.1 Contributions ...... 143 10.1.1 Interval arithmetic ...... 143 10.1.2 Interval constraints ...... 143 10.1.2.1 Propagation ...... 143 10.1.2.2 Relational box-consistency ...... 144 10.1.2.3 Interval constraints with exploration ...... 144 10.1.3 Implementation ...... 145 10.2 Future work ...... 145 10.2.1 Dependency effect ...... 146 10.2.2 Parametric CSPs ...... 146 10.2.3 Criteria for choosing a consistency algorithm ...... 147 10.2.4 Confidence quantification ...... 147 10.2.5 Disjunctive constraints ...... 148 10.2.6 Iterative widening with exploration ...... 148

Bibliography 149

Appendix A Source Code 154 A.1 Box consistency ...... 154 A.2 Adaptive box consistency ...... 155 A.3 Dynamic box consistency ...... 155 A.4 Box satisfaction algorithms ...... 156 A.5 Default explorer and explorer with propagation ...... 159 A.6 Interval constraint global optimization solver with exploration ...... 159 List of Tables

Table 9.1 Propagating and squashing in an interval constraint solver. The unit of Duration is seconds...... 123 Table 9.2 Comparison between BC, ABC, and DBC. The unit of Duration is

microseconds...... 124

¿   Table 9.3 Performance of MS¾ , MS , MS , and MS . The unit of D is seconds. . 135 Table 9.4 Comparing ExplLB with squashLftC...... 137 List of Figures

Figure 1.1 A flow chart indicating chapters’ prerequisites...... 9

Figure 2.1 Left bound squashing algorithm using interval arithmetic...... 19 Figure 2.2 A squashing algorithm that uses interval arithmetic to obtain func-

tional box consistency...... 20 ´µ

Figure 3.1 The algorithm MS ¼ . The function atomic specifies whether its box

argument is atomic or not...... 25

Figure 3.2 The algorithm MS ½ . The function has as value the width of its

interval argument. The tolerance is assumed to be sufficiently large as termination of the algorithm is not assured otherwise...... 27

Figure 3.3 The algorithm MS ¾ . In case of success, an interval of the desired width has been found. Otherwise, a lower bound for the global minimum

has been found that is the best possible one, given the expression for and the precision of the arithmetic...... 29

Figure 3.4 The algorithm MS ¿ . In case of success, an interval of the desired width has been found. Otherwise, an interval for the global minimum has

been found that is best, given the expression for and the precision of the arithmetic...... 33

Figure 3.5 The algorithm MS  . After termination, the best outer approximation

to the Æ -minimizer is the set of boxes left in the cover...... 35 List of Figures xiii

Figure 3.6 A terminal situation of the algorithm MS outer approximation for

the Æ -minimizer...... 36

Figure 3.7 The algorithm MS  . After termination, the set of boxes left in the

· Æ µ cover is the best inner approximation for the ´ -minimizer...... 38

Figure 3.8 A terminal situation of the algorithm MS for inner approximation

· Æ µ

of the ´ -minimizer...... 39

´µ Figure 4.1 The algorithm . The function atomic specifies whether its box argument is atomic or not. It finds an inner and an outer approximation

for System (1.2)...... 42 ´µ

Figure 4.2 The algorithm FS Ç . The function atomic specifies whether its box

argument is atomic or not...... 44 ´µ

Figure 4.3 The algorithm FSÁ . The function atomic specifies whether its box argument is atomic or not...... 46

Figure 4.4 The algorithm FS. The function atomic´µ specifies whether its box argument is atomic or not...... 48 Figure 4.5 An improved version of the algorithm FS. This version avoids pro- cessing boxes for which the parents are proved to satisfy some of the con- ditions involved in the constrained global optimization problem...... 50 Figure 4.6 The algorithm FPFS. In case of success, an interval of the desired width has been found. Otherwise, an interval for the global minimum has

been found that is best, given the expression for and the precision of the arithmetic...... 52

Figure 4.7 The algorithm FPFS½ . In case of success, an interval of the desired width has been found. Otherwise, an interval for the global minimum has

been found that is best, given the expression for and the precision of the

arithmetic. So this information can be forced by setting to zero...... 55 List of Figures xiv

Figure 4.8 The algorithm LPOFS. After termination, the set of boxes left in the Æ cover is the best outer approximation to the -minimizer...... 56

Figure 4.9 The algorithm LPIFS. After termination, the set of boxes left in the

´ · Æ µ cover is the best inner approximation to the -minimizer...... 58

Figure 5.1 A pseudo-code for GPA...... 63

Figure 5.2 A system of non-linear inequalities without multiple occurrences

Ò

of variables. Instead, the set ½ is partitioned into equivalence

Ô  ½ Ò ½ Ò

classes ½ where is a subset of , for

½ Ð ÐÕ .An constraint asserts that its arguments are equal. . . 64 Figure 5.3 Hardware model for a CSP...... 66 Figure 5.4 Left bound squashing algorithm using interval constraints...... 69

Figure 6.1 A propagation algorithm based on recursive functions on power sets. 75 Figure 6.2 Dual propagation algorithm based on recursive functions on power sets...... 76 Figure 6.3 The dual generic propagation algorithm...... 77 Figure 6.4 A clustered propagation algorithm based on recursive functions on power sets...... 78 Figure 6.5 Architecture of our implementation...... 81 Figure 6.6 UML diagram for our implementation...... 82 Figure 6.7 A graphical user interface for our implementation showing the mod- eling (clusters of constraints) and the intervals resulted from applying prop- agation on the user’s CSP...... 84

Figure 7.1 Illustration of BC steps...... 86 Figure 7.2 Illustration of DBC steps...... 87 Figure 7.3 Illustration of the boundary intervals removed by BC...... 87 Figure 7.4 Illustration of the boundary intervals removed by DBC...... 88

Figure 7.5 Structure of dynamic box consistency algorithm with a tolerance .90 List of Figures xv

Figure 7.6 A pseudo-code for adaptive box consistency algorithm with a toler-

ance ...... 92 Figure 7.7 An improved box-consistency algorithm...... 94

Figure 8.1 A visual for the steps involved in interval constraints with exploration.100

Figure 8.2 Computing a lower bound for Ñ
Ò using exploration...... 102 Figure 8.3 Architecture of interval constraints with exploration for global opti- mization...... 104 Figure 8.4 Interval constraint algorithm with exploration. The function reduce

applies propagation or squashing and makes use of the function explLB.It returns the reduced domain of , and the interval of . The function explore

explores the optimization problem, and returns a point , and a real number

´ µ

Ü . The function returns the width of the largest component of . . 106

Figure 8.5 Left bound stretching algorithm using interval constraints...... 112

´ µ Figure 8.6 Zoomin algorithm using interval constraints. The function

gives the width of the largest component of ...... 113

Figure 8.7 Zoomout algorithm using interval constraints...... 114

Figure 8.8 An algorithm for proving the existence of solutions of  ...... 115

Figure 8.9 -slicing algorithm...... 116

Figure 9.1 Architecture of BelSyste...... 118

Figure 9.2 Components of BelSyste and their dependencies...... 120

¾

 ½ ¼¼ ½¼

Figure 9.3 The graph of , where and in

½¼ ¼¼

 , using BelSyste...... 128

¾ ¾ 

 ¼ ´ ½µ · ½¼  ¼

Figure 9.4 The graph of and , where

¼ ¼ ½ ¼ ½ ¼

and in , using BelSyste...... 141

¾ ¾ 

 ¼ ´ ½µ · ½¼  ¼

Figure 9.5 A graph for with

¼ ¼ ½ ¼ ½ ¼ and in using BelSyste...... 142 List of Abbreviations

= the greatest lower bound. ABC = Adaptive box consistency algorithm. BC = Box consistency algorithm. CSP = Constraint satisfaction problem. DBC = Dynamic box consistency algorithm. GOP = Global optimization problem. ICSP = interval constraint satisfaction problem.

Notations

The feasible region.

The upper bound, found so far, for the global minimum.

The set of floating-point numbers. Ê

The set of real numbers.



The ICSP representing the feasible region .

The interval arithmetic evaluation of associated with a given expression for .

A function that takes a set of reals and gives the smallest floating-point interval

containing .

Ö

A function that takes a set of reals and gives the smallest real interval containing

.

The objective function.

Notations xviii

Ñ
Ò The global minimum.

I The set of floating-point intervals.

IÊ The set of real intervals. Glossary

Box A Cartesian product of intervals.

Box consistency Squashing all the components of a box until no narrowing is possible.

Box satisfaction Computing a box that cannot be stretched to contain solutions and only solutions.

Box unsatisfaction Computing a box that cannot be stretched to contain no solutions.

Constrained global optimization A global optimization problem for which the feasible region is not a box.

Cover for A set of non-empty subsets of such that their union is .

Cover in

A set of non-empty subsets of .

Dependency effect The dependency effect is the excessive over estimation of the range of a function that

arises when at least one of the variables occurs more than once in the expression of

½ ½ ¼ ¼

the function. For example, the expression gives , and not , when

 ¼ ½ .

Elementary operations

Glossary xx

Simple operations such as · .

Endian Computer architectures are characterized as big-endian or little-endian depending on how memory cells are arranged.

Exploration The technique of using explorers to improve the speed of interval constraints.

Explorer A polynomial time algorithm that extracts informal information about the problem considered.

Feasible point A point that satisfies all the conditions in a constrained global optimization problem.

Functional box consistency Obtaining box consistency using interval arithmetic.

Inner approximation for a -minimizer

A set of boxes that are subsets of the -minimizer.

Iterative widening It starts from a point near a solution and tries to find a box that may contain a solution.

-minimizer

The set of points for which the objective function is within of the global minimum.

Minimizer The set of points for which the objective function attains the global minimum.

Outer approximation for -minimizer

A set of boxes whose union encloses the -minimizer.

Relational box consistency Obtaining box consistency using interval constraints. Glossary xxi

Reliable Qualifies any algorithm with guaranteed accuracy. A reliable algorithm usually re- turns a set of boxes containing the solutions of the problem at hand. The results of a reliable algorithm are said to be reliable.

Rounding mode To control the round-off errors of the arithmetic operations, the IEEE standard 754 specifies four rounding modes: to the nearest, to zero, to plus infinity, and to minus infinity.

Squashing/Narrowing/Pruning Removing from the sides of a box the parts that are proved not to contain any solution.

Stretching Enlarging a box to contain more solutions and only solutions.

Unconstrained global optimization A global optimization problem for which the feasible region is a box. Acknowledgement

I cannot thank enough my supervisor, Professor Maarten van Emden, for accepting me to be his student. Coming from an educational system in which theories are put first, I was very lucky to be under his supervision. “Hello Belaid, I found this interesting puzzle, ...”, with a smile in his face, this is how Professor van Emden starts when I drop by his office. Any time I meet him, he makes my day by enriching my knowledge. Full of ideas, full of patience, full of support, full of respect, .... Indeed, what a wise teacher you are. Without your help, your support, your encouragements, and your financial aid, I never would have completed the program. Without your technical discussions, suggestions, and enthusiastic supervision this thesis could not have been completed. I would like to warmly thank my co-supervisor, Professor Bill Wadge, for his insights, comments, and suggestions regarding my thesis. Many thanks to Professor John Ellis, Professor Fayez Gebali, and Professor Mohamed Ibnkahla, for taking the time to read over the thesis, for their comments, and for their encouragements. I would like also to thank my colleagues in the department of computer science for their help and support. I am very grateful to my wife, Ghaida Hammado, for her love and support. I am also very grateful to all my friends and my teachers for their help during the course of my research. Finally, I am forever indebted to my mother who taught me how to be strong, and to my father who showed me how to be patient. Dedication

To My Parents To My Wife To My Teachers To My Friends Chapter 1

Introduction

This chapter is intended to provide a general overview of the global optimization problem.

1.1 Motivation

Nowadays, optimization seems to be a necessity in almost every field. Various disciplines, from business to engineering, require some sort of optimization in order to improve the performance of every day tasks. To achieve this goal, we need to develop a mathematical model consisting of a numerical objective function that depends on numerical decision variables that are subject to certain conditions. The global optimization problem is to find the values of the decision variables, which satisfy the conditions, and minimize not only locally but also globally the objective function.

A photon minimizes the time traveled between two points. More generally, the varia- tional principle, from which several laws of physics can be derived and which states that the trajectory of a particle is the one that minimizes a certain quantity called action (see

Chapter ½ in [11]), is an instance of the global optimization problem. In statistical me- chanics, molecules tend to be distributed in a way that maximizes their entropy. A taxi driver minimizes the distance between his position and the destination. An engineer, when faced with many design options, tries to choose the optimal one. In any approximation, the error between a proposed model and a set of data must be minimized. The list of examples where optimization is involved is endless. As claimed in [37], almost every mathematical 1.2 The problem statement 2

problem can be transformed into an optimization problem. This betrays of course a narrow view of mathematics. But it is true that surprisingly many problems can be so transformed.

For the more realistic mathematical models, the global optimization problem is difficult to solve. Faithfully modeling real systems introduces objective functions with an unknown and large number of local minima, nonlinear conditions, and a mixture of discrete and continuous decision variables. In the presence of these attributes, numerical analysis has no satisfactory method to solve the global optimization problem. For a long time, it has been thought that no numerical method could guarantee solving the global optimization problem in the presence of nonlinearity [56]. The reasoning behind such a thought was based on the finite precision of computers: there is no way to faithfully approximate the values of a function, especially when it is nonlinear, using a finite number of points. This argument is true when the function to be optimized is evaluated at sampled points, but when it is evaluated over a range the argument does not hold. Indeed, inter- val arithmetic and interval constraints are both capable of returning a satisfactory interval containing the global optimum.

1.2 The problem statement

Informally, the global optimization problem deals with finding the greatest lower bound, when it exists, of a given real-valued function in a set described by certain conditions.

Formally, the global optimization problem is to find

´µ

(1.1)

Where stands for the greatest lower bound, the objective function is a real-valued

Ò Ò

Ê Ê function from Ê to , and , often called the feasible region, is a subset of . The

greatest lower bound of , when it exists, is called the global minimum and is denoted by

Ò

Ê Ê

Ñ
Ò . The symbol denotes the set of real numbers, and is the Euclidean -space. If is a Cartesian product of intervals, then the global optimization problem is said to be unconstrained, otherwise it is constrained. 1.2 The problem statement 3

The restrictions we impose on are the following: For the use of interval arithmetic we suppose that has an expression containing

operations for which we have interval extensions. For the use of interval constraints we suppose that can be decomposed into con- straints for which domain reduction operators are available.

These concepts are covered in the upcoming chapters.

Ò Ê The set is a subset of described by a system of linear and nonlinear inequalities similar to System (1.2). Without loss of generality, we consider the generic system of both linear and nonlinear

inequalities below:

´ µ ¼

½ ½ ¾ Ò

´ µ ¼

½ ¾ Ò ¾ (1.2)

......

´ µ ¼

½ ¾ Ò

 ½

Where for every ,
has the same restrictions as . That is, in the case of

interval arithmetic we suppose that
is defined by an expression containing operations for which interval extensions are available, and in the case of interval constraints,
can be decomposed into constraints for which domain reduction operators exist.

The optimization problem can be decomposed into the following subproblems: the fath- oming problem and the localization problem.

Fathoming problem This is concerned with finding what the global minimum is. That is, we compute the value Ñ
Ò (or an interval that contains it), and we are not concerned

about the values of at which such a value occurs.

Localization problem In the localization problem, we are concerned with locating where the global minimum occurs. 1.3 Methods for solving global optimization problems 4

That is, finding the minimizer, which is defined as

´µ 

Ñ
Ò and .

Due to machine limitations, the minimizer cannot always be computed. Instead, however,

¼

we compute, for a suitable , the -minimizer defined as:

´µ

Ñ
Ò and .

We call special attention to the fact that we are dealing with global optimization and not local optimization. In local optimization only the determination of local minima is desired.

1.3 Methods for solving global optimization problems

The global optimization methods can be classified into the following two main categories:

1. point methods, and

2. bounding methods.

The point methods are the classical methods for global optimization, that include deter- ministic and stochastic techniques. They are based on computing function values at sam- pled points. Most literature about such methods covers one of these methods: Newton-like methods [12, 13], Clustering methods [33, 56], Evolutionary Algorithms [17, 38], Simu- lated Annealing [34], Statistical Methods [43], Tabu Search [16], and hybrid methods that combine some of these (see for example [39]). The similarity among all these methods is that none of them guarantees the correctness of its results. In fact, these methods may not terminate or may terminate with a result far from the global minimum. For an objective function with a large number of local minima, even the best point method can fail to spot any value in the minimizer and, in general, may get trapped in a local minimum that is not the minimum. On the other hand, the bounding methods compute an interval that is proved to contain the global optimum. These methods can be classified into one of the following classes depending on the way the bounds of the interval are computed: 1.3 Methods for solving global optimization problems 5

1. Lipschitzian Approach. This approach is based on decomposing the box on which

the function is to be bounded into boxes in which the function is Lipschitzian. A

¼

function is Lipschitzian in a box if and only if there exists a constant , such

that for any

´µ ´ µ

If the value of the function at point is given, then an interval containing the range

of is

´ µ ´ µ ·   ,

where

 Ñ Ü

.

ÙÚ ¾

2. Interval Arithmetic. This approach uses interval arithmetic evaluation to obtain the bounds of on a given box . This evaluation is based on the notion of interval extension defined in the next chapter.

3. Interval Constraints. In this approach, the bounds of the function are computed based on applying the constraint propagation algorithm on a “moving” constraint system. This will be explained later.

The common characteristic of all bounding methods is that they can be implemented so that the computation is a proof of the result. Their power in solving any global optimization problem of the form (1.1) relies on bounding the value of the global minimum between two floating-point numbers. The global minimum is proved to be found between these two numbers. Thus, the qualification “bounding”. One interesting property of interval methods is that they return bounds even in cases

where it is difficult to compute a point in the feasible set . For example, in such cases, the

Ò Ê interval given by the interval arithmetic evaluation of over is an interval containing the global minimum. Because such an interval is in general too wide, we will suppose

that is specified by a system of equation similar to System (1.2) in order to get sharper bounds for the global minimum. 1.4 Layout of the dissertation 6

Since we are more interested in reliable computing, we will focus only on the bound- ing methods: optimization is so important that we must tackle it correctly, completely, and efficiently. Among bounding methods, we will only treat interval arithmetic and inter- val constraint methods. Even though they are implemented in several software packages (UniCalc, Filib++, Numerica, etc), they still have certain aspects that can be improved to enhance their performance. This dissertation gives an overview of the deficiencies in these approaches, suggests improvements and provides new approaches that can be used to tackle the global optimization problem in a more time-efficient way.

1.4 Layout of the dissertation

As we pointed out earlier, the bounding methods constitute the main topic of the disserta- tion, and they are presented in two parts. The first part is titled “global optimization using interval arithmetic”. In this part, we discuss interval arithmetic and its use to achieve global optimization. More precisely, interval arithmetic evaluation is used to implement the fathoming algorithm, the inner- approximation algorithm and the outer-approximation algorithm. These algorithms are derived from the Moore-Skelboe algorithm in a rational reconstruction manner similar to that of Lakatos [35]. The second part deals with global optimization using interval constraints. After reviewing the necessary notions of interval constraints, we present revised propa- gation algorithms. These algorithms are important in that they improve the time-efficiency of the usual propagation algorithm. As we will emphasize in this part, by a suitable choice of a “moving” constraint system, the global optimization problem can be solved by solving constraint satisfaction problems. In fact, as shown in [50], there is no difference between the global optimization problem and solving constraint satisfaction problems; they are even in the same complexity class. Another important aspect of interval constraints is pruning. New approaches for prun- ing are discussed and compared to box-consistency, and bound-consistency algorithms. We 1.4 Layout of the dissertation 7

fully describe the idea behind dynamic box-consistency and adaptive box-consistency al- gorithms. These algorithms are used to solve some constraint satisfaction problems as well as some global optimization problems.

In Chapter  we describe a new approach for tackling global optimization problems. We then conclude by presenting some experimental results, and by mentioning our contri- butions into the field of global optimization. The rest of the dissertation is organized as follows:

Part I: Global optimization using interval arithmetic This part presents interval arithmetic and its use in global optimization. It is com- posed of the following chapters:

1. Chapter ¾: Interval arithmetic This chapter serves to provide the necessary background and to establish the terminology used in interval arithmetic.

2. Chapter ¿: Unconstrained global optimization via interval arithmetic In there, we discuss Moore-Skelboe algorithm and its improvements. These improvements are implemented and used to solve several benchmarks.

3. Chapter : Constrained global optimization This chapter shows how interval arithmetic is used to solve constrained global optimization problems. As we will argue later, ensuring that the conditions of the global optimization problem are satisfied is difficult with interval arithmetic compared to interval constraints. This will lead us into interval constraints and their power.

Part II: Global optimization using interval constraints In this part we tackle global optimization problems using interval constraints.

1. Chapter : Interval constraints This chapter serves to provide the necessary background and to establish the terminology used in interval constraints. 1.4 Layout of the dissertation 8

2. Chapter : Dual, clustered, deterministic, and selective propagation algo- rithms In this chapter we discuss how to improve the propagation algorithm. Two algorithms are presented: dual propagation and propagation by selective initial- ization.

3. Chapter : Dynamic box-consistency and adaptive box-consistency In this chapter, we present two new algorithms for pruning: dynamic box- consistency and adaptive box-consistency, and contrast them with the existing box-consistency algorithms.

4. Chapter : Global optimization via interval constraints In this chapter, we show how interval constraints can solve both constrained and unconstrained global optimization problems. ”Exploration” is used to improve

the efficiency of interval constraints.

Chapter : Experimental results

In this chapter, we present the experimental results of our implementation. ½¼ Chapter : Conclusion We conclude the dissertation with our main contributions in the field of global opti- mization as well as some suggestions for future work.

The flow chart in Figure 1.1 indicates the prerequisites of each chapter. 1.4 Layout of the dissertation 9

Chapter 1 Introduction

Part I Part II

Chapter 2 Chapter 5 Interval arithmetic Interval constraints

Chapter 3 Chapter 6 Chapter 7 Unconstrained global optimization Dual, clustered, deterministic, and Dynamic box-consistency and via interval arithmetic selective propagation algorithms Adaptive box-consistency

Chapter 4 Chapter 8 Constrained global optimization Global optimization via interval via interval arithmetic constraints

Chapter 9 Experimental Results

Chapter 10 Conclusion

Figure 1.1. A flow chart indicating chapters’ prerequisites. Chapter 2

Interval Arithmetic

In this chapter we shall cover briefly the main concepts of interval arithmetic and fix the terminological and the notational conventions used in the next chapters. For a more com- prehensive presentation of interval arithmetic, the reader may refer to [32, 46, 49, 20, 2].

2.1 From floating-point numbers to interval arithmetic

Computers can only manipulate a finite amount of information. This finite aspect pre- vents them from representing most of the real numbers. In fact, the set of real numbers is uncountable and the set of numbers that can be processed by a computer is finite. To per- form numerical computations involving real numbers, the IEEE 754 standard [1] defines floating-point numbers, and describes the operations associated with them. More precisely, the standard specifies: number formats, basic operations, conversions, exceptional condi-

tions, and rounding modes (round to nearest, round to zero, round to , and round to

· ).

In conventional methods, a problem Ö involving real numbers is transformed, through a

process called rounding, into a problem involving floating-point numbers. The solutions

Ö of are generally taken as a satisfactory approximation to the solutions of . But, as proved in practice, this is not always the case. Due to the fact that an operation over floating-point numbers can yield a real number that is not a floating-point number, there are many problems where the accumulation of repeated rounding errors affect the final results.

2.2 Advantages of interval arithmetic 11

To overcome this, we enclose the problem Ö in a problem involving representable

ranges of real numbers. Using suitable roundings, the solutions of are proved to contain those for Ö . An interesting consequence of this technique is that instead of returning a floating-point number that, more or less, approximates an exact solution, we return a range containing it. Thus, numerical problems can be solved while bounding errors from all sources, including rounding errors, modeling errors and data uncertainty. This technique is interval arithmetic.

2.2 Advantages of interval arithmetic

Interval arithmetic was introduced to control errors, but that’s not its only advantage; there are many more. Below we only and briefly present a few. In interval arithmetic there is no need for exception handling, which is complicated

in general. For instance, division by zero, which is considered an error in conventional

· methods, generates an interval of the form  in interval arithmetic. Moreover, as we will show in the last chapter, the overflow that occurs in integer operations can be eliminated by means of interval arithmetic. Interval arithmetic increases the opportunity for parallelism [37]. In fact, several al- gorithms based on interval arithmetic are parallel in nature, because they proceed by re- cursively decomposing regions into subregions and deleting the regions that are proved to contain no solution. Interval arithmetic is also a very useful tool for sensitivity analysis. In fact, the beauty of computing with intervals is that it makes accuracy visible as the program progresses, thus, increasing the chances for producing better models. More generally, interval arithmetic provides a simple framework for evaluating different mathematical models. Another interesting property of algorithms that are based on interval arithmetic is that they are widely applicable. As opposed to conventional algorithms, which are developed for a restricted class of problems, interval arithmetic algorithms can be used for a wide class of problems. 2.3 Basics of interval arithmetic 12

A surprising implication of the use of interval arithmetic is the flexibility it offers in engineering systems. For instance, suppose that a system is specified by a certain number of unknown parameters (e.g. capacity, resistance, etc.) that satisfy certain conditions. A conventional method will only return a single value for each parameter. Because the values of these parameters must adhere to a certain standard, one of the computed values may not be available, and, thus, causing the system to be unrealizable. An interval method, on the other hand, returns an interval for each parameter. Obviously, the chances to find practical values in such intervals are greater. From our perspective, however, the most striking consequence of interval arithmetic is the concept of numerical provability. Using interval arithmetic, several equations can be proved to have either a solution or not. The graph of a function can be proved to be enclosed in a set of rectangles displayed on a computer screen [58, 40] (see the last chapter for an example). Safety simulations can be done correctly on finite precision machines [40].

2.3 Basics of interval arithmetic

2.3.1 Intervals

In what follows, we briefly review some definitions on floating-point intervals. We refer

the reader to [28] for more detail.

Let Ê be the set of real numbers, and let be the set of floating-point numbers. Let

be the greatest finite positive floating-point number.

·

For every finite floating-point number in , let denote the smallest floating-point

number greater than , and the greatest floating-point number less than . Note that if

·

  ·  

, then , and if , then . Let be any floating-point

·

   · 

number. If , then , and if , then . By convention, if

·

 ·  ·   , then , and if , then .

Given a real number , the greatest floating-point number that is smaller than or equal

to is denoted by , and the least floating-point number that is greater than or equal to is denoted by . 2.3 Basics of interval arithmetic 13

A real interval is a closed connected subset of reals. In other words, it is a subset

    · 

of Ê that has one of the following forms , ,

   ·  Ê

 ,or , where the bounds and are real numbers. The

set of real intervals is denoted by IÊ . When the bounds and are restricted to the set of finite floating-point numbers, we obtain floating-point intervals. The set of floating-point

intervals is written as I .

A box is a Cartesian product of intervals. The elements of a box are called points.A

 Ò

point of a box ½ is specified by real numbers called the components

of . The ith component of must be an element of
. The point can be thus denoted

´ µ

Ò

by ½ , where is the ith component of .

     ´ µ

½ Ò Ò ½ Ò

Given a box ½ , the points of the form , where

½

is either
or for every in are called the corners of the box .Anon-corner point of is a point in that is not a corner. From now on, when we write “interval” without qualification we mean a floating-point

interval.

   ´ µ ´ µ

For every interval , we use lb for the left bound , rb for the right

·

´ µ   

bound , and for the width of . The intervals of the form , and



 where is a floating-point number, are called atomic or canonical intervals. Any



interval of the form  is said to be degenerate.

Ê ´ µ Ê

For a given subset of , we denote by Ö the smallest real interval in I contain-

´ µ ing , and by the smallest floating-point interval in I containing . A notion that will be used extensively in the next two chapters is the notion of cover. It

is defined as follows.

¼ Æ

Let be an integer. If ½ is a set of non-empty subsets of a set , then

Æ ½

the set is called a cover in . If, in addition, the union of ¼ is , then the set

Æ ½ ¼ is a cover for . 2.3 Basics of interval arithmetic 14

2.3.2 Interval extensions and Inclusion functions

· Let be one of the binary operations . The operation could be extended to

real intervals as follows

 

and such that ,

where and are two real intervals. But because the right hand of the equation above

½½ ½   ½ ½ · may not be a single real interval (for example, ½ ), we

instead use

 ´  µ

Ö and such that .

This extension is called real interval extension of .

As an example, the real interval extensions for · and are expressed respectively as

·   ´ µ · ´ µ ´ µ · ´ µ   ´ µ ´ µ ´ µ ´ µ

lb lb rb rb and lb rb rb lb

for every pair of real intervals and .

µ The real interval extensions of the elementary functions ´ can be defined similarly.

A floating-point interval extension of the binary operation is defined as

 ´  µ

and such that ,

where and are two floating-point intervals.

As an example, the floating-point interval extensions for · and are respectively given

·   ´ ´ µ · ´ µµ ´ ´ µ · ´ µµ   ´ ´ µ ´ µµ

by lb lb rb rb and lb rb

´ µ ´ µµ

´rb lb for every pair of floating-point intervals and . An easy way to

·

numerically compute the bounds of the interval above is to set the rounding mode

´ µ· ´ µ · ´ µ·

to just before performing lb lb , and then to before performing rb

µ rb´ . The same thing can be done when computing the bounds of . The way of setting the rounding modes just described is usually called outward rounding. The reader may refer to [28] for an implementation of the other operations.

2.3 Basics of interval arithmetic 15

µ The floating-point interval extensions of the elementary functions ´ can be defined similarly. In what follows, “interval extension” will be used to mean floating-point interval exten-

sion.

Ò Ò

Ê Ê Given a function from to ,aninclusion function of is a function from I to

I such that

Ò

´ µ ´ µ

I ,

´ µ ´ µ  ´µ

where is the range of over , that is, .

Ò

´ µ  ´ ´ µµ

If, in addition, for every I we have , then is said to be minimal.

 ´ µ ´ µ

Ò ½ Ò Given a box ½ , we use to mean . For a large class of functions, interval extensions turn out to provide a simple framework

for constructing their inclusion functions. Indeed, given an expression for a real-valued

Ò

´ µ Ê Ê Ò

function ½ from to involving the elementary operations mentioned ear-

´ µ Ò

lier, an inclusion function ½ can be constructed by replacing, in the expression

of , each elementary operation by its extension and each variable
by the interval .

Ò Moreover, if each of the variables ½ occurs at most once in the expression of ,

then is minimal [23, 32, 46]. By interval arithmetic evaluation of we mean the inclusion function we just de-

scribed. Note that the result of the interval arithmetic evaluation of a function depends on the expression chosen for . The fact that the interval arithmetic evaluation of is an

inclusion function of is commonly known as the fundamental theorem of interval arith- metic. For a formal way of treating expressions, functions, and interval arithmetic evaluations, the reader can refer to [63].

To obtain an interval that better approximates the range of , we often make use of

monotonicity defined below.

Let be a function from a set of sets to a set of sets. The function is said to be

2.4 Solving inequalities via interval arithmetic 16

´ µ ´ µ

¾ ½ ¾ ½ ¾

monotonic iff for every ½ , and in such that ,wehave .

Ò

Ê Ê

As a special case, we have the following. Let be a function from to , and let be

Ò ½ Ò

an inclusion function for . is monotonic iff for all intervals ½ and ,

´ µ ´ µ

½ Ò Ò ½ Ò ½ Ò such that ½ ,wehave

If is a function with an expression involving elementary operations, then the interval

arithmetic evaluation associated with the expression of is monotonic. From now on, the notation will denote the interval arithmetic evaluation of with respect to a given

expression of .

We call special attention to the fact that the interval arithmetic evaluation of a function

depends on the expression for . Moreover, the interval arithmetic evaluation of a function is not minimal in general, because of the dependency effect, an effect that can be described as follows. The dependency effect arises when at least one of the variables occurs more

than once in the expression of . To illustrate such this effect, consider the function of one

´µ 

variable defined by the expression . Based on this expression, the interval

´ µ   ¼ ½ ´¼ ½µ 

arithmetic evaluation of is therefore .For we have

½ ½

 . Thus, the function is not minimal. It is easy to verify that the minimal interval

´ µ  ¼ ¼ extension of is a constant function given by .

2.4 Solving inequalities via interval arithmetic

We are now in a position to analyze the relation between interval arithmetic and solving inequalities.

Without loss of generality, let us consider the following generic inequality:

´ µ ¼

¾ Ò

½ (2.1)

¾ Ò ½ ¾ Ò where the variables ½ are respectively in the intervals , and

where the function has an expression involving the elementary operations mentioned in

the previous section.



¾ Ò

Suppose that is the interval arithmetic evaluation of over ½ , i.e.

´ µ

¾ Ò ½ , where is the interval arithmetic evaluation of associated with the 2.4 Solving inequalities via interval arithmetic 17

expression of . The most useful fact about interval arithmetic is that, with only this information, we

can already state three important observations about Inequality (2.1).

´ µ ¼

¾ Ò

1. If lb , then Inequality (2.1) has no solution in the box ½ .

´ µ ¼

¾ Ò 2. If rb , then Inequality (2.1) has the whole box ½ for

solutions.

µ ¼ ´ µ ¼

3. If lb´ and rb , then Inequality (2.1) may or may not have a solution in

¾ Ò the box ½ .

The last case deserves further study. In fact, from our point of view, it is the most interesting

case. Here is a complete list of instances that can happen with this case.

¾ Ò 1. If the intervals ½ are all degenerate, then, as far as interval arithmetic

is concerned, we cannot decide whether the Inequality (2.1) has a solution or not.

¾ Ò 2. If at least one of the intervals ½ is canonical and the others are canoni-

cal or degenerate , then, as far as interval arithmetic and the expression of are con-

¾ Ò cerned, we cannot decide whether a non-corner point in the box ½

is a solution or not. We emphasize that such a claim is based on a given expression for . Using another expression may yield a different outcome. However, if is minimal, then, as far as interval arithmetic is concerned, there is no way to overcome such an undecidability. The corners of the box can be checked separately.

3. If we have none of the above, then, because of monotonicity, we need to split the box

¾ Ò ½ into smaller boxes and look for solutions in each one of them.

In this section, we addressed the question of finding solutions of an inequality using interval arithmetic. Since searching for all solutions often produces an exponential running time, an interesting alternative is to find, based on interval arithmetic, a small box that encloses the solutions of an inequality. This we do next. 2.5 Functional box consistency 18

2.5 Functional box consistency

Ò

Without loss of generality, let us consider Inequality (2.1). Let ½ be intervals.

´ µ Ò

We denote by the interval ½ .

µ ¼

As we mentioned in the previous section, if rb´ , then the points in the box

Ò ½ Ò ½ are solutions to Inequality (2.1). Hence, the box is the

smallest box containing the solutions to Inequality (2.1).

´ µ ¼ Ò

If lb , then there is no solution for Inequality (2.1) inside the box ½ .

´ µ ¼ ´ µ Ò If, however, lb rb , then it may be possible to make the box ½

smaller by reducing one of the intervals while keeping the others fixed. In fact, assume that

´ µ ½

for some floating-point number ½ with lb ,wehave

´ ´ ´ µ  µµ ¼ Ò

lb lb ½

 ´ µ ½ In this case, the interval ½ can be improved to rb . This process of reducing intervals is called squashing, pruning,orbox narrowing [31, 8, 18, 27, 7, 60]. In qualitative terms, squashing can be viewed as a reduction operator that shrinks a box by removing, from its sides, the parts that are proved not to contain any solution.

Before proceeding, we have to make a brief digression to define the notion of functional

box consistency.

Inequality (2.1) is said to be functionally box-consistent with respect to
iff

 ´ ¼ ´ ´ µ µµ

½
½

·½ Ò

.



Inequality (2.1) is said to be functionally box-consistent with respect to the box ½

 ½
Ò iff it is functionally box-consistent with respect to each for every .

The usual way to achieve functional box consistency is by shrinking the left and then

the right bound of each interval. This process is repeated for each
until no interval can be further pruned. The function that shrinks the left bound of the interval
is given in Figure 2.1. A similar function can be used to narrow the right bound of the interval

2.5 Functional box consistency 19

 µ

Floating-point number function squashLftI´

´ ´   µµ ¼

½
·½ Ò

if (lb ½ ), then return ;

if ( ) return ; //[a,b] is canonical

//[a,b] is not canonical, so has midpoint

  

midpoint of ;

¼

 ´  µ

squashLftI ;

¼ ¼ ¼

if ( ) return ;// is the best left bound

//left bound was improved all the way up to

//so maybe m can improved some more

 µ

return squashLftI´ ;

Figure 2.1. Left bound squashing algorithm using interval arithmetic.

. We call it squashRghtI. Both functions are used to squash the whole box as shown in

Figure 2.2.

 Because the process of pruning becomes slow as the interval  gets smaller, Granvil- liers, Goualard and Benhamou proposed what they call weak box consistency [18]. This

consistency is similar to box consistency explained above, except that the squashing pro-

 cess stops when the width of  is smaller than a predetermined value . As we shall see later, both consistencies can be improved even further using dynamic box-consistency and adaptive box-consistency algorithms.

2.5 Functional box consistency 20

´ µ

¾ Ò

Box function squashBoxI ½

½

initialize the set to ;

while ( is not empty)

Ò

//the box ½ can perhaps be reduced further remove from ;

lftDone=false; rghtDone=false;

//squash the left and the right bounds of

while(lftDone rghtDone)

if(lftDone)

´ µ  ´ µ
lb
squashLftI ;

lftDone=true;

´

if (lb
is reduced) rghtDone=false;

if(rghtDone)

´ µ  ´ µ
rb
squashRghtI ;

rghtDone=true;

´

if (rb
is reduced) lftDone=false;

//prune the right bound of the interval

if (
is reduced)

 ½ 

add every and to ;

Ò //the box ½ cannot be reduced any further

//the functional box consistency is thus achieved

Ò

return ½ ;

Figure 2.2. A squashing algorithm that uses interval arithmetic to obtain functional box consistency. Chapter 3

Unconstrained global optimization via interval arithmetic

We are now in a position to tackle the unconstrained global optimization problem in the light of interval arithmetic. After briefly presenting Moore-Skelboe (1974), Ichida-Fujii (1979), Hansen’s (1980), and Hansen and Sengupta (1983) algorithms, and after revealing their weaknesses, we shall present new improvements of these algorithms. More precisely, we shall discuss how the fathoming algorithms settle the fathoming problem, and how the inner-approximation and the outer-approximation algorithms handle the localization problem. We shall also present theorems that support our algorithms.

Throughout this chapter, we consider the problem (1.1), where the objective function

Ò

Ê can be evaluated in interval arithmetic, that is exists, and where the set is a box in .

3.1 Motivation

Interval arithmetic algorithms for global optimization are examples of a larger group of algorithms, referred to as branch and bound algorithms. Branch and Bound applied to interval computations is a general search method that is best viewed as an iteration of two

steps: The bounding step uses computed bounds to discard subboxes of . 3.1 Motivation 22

The branching step generates a set of smaller subproblems by dividing boxes that weren’t discarded in the bounding step.

The subproblems created in the branching step are then tackled until all solutions are found.

Interval arithmetic algorithms for global optimization are based on the same idea. They

use interval arithmetic to obtain a left and a right bound for on a given box. Depending on the size of the intervals that we provide as arguments, we might end up with a wide

interval, especially in the presence of the dependency effect. As a result, we subdivide the

box into small boxes and then evaluate over each of these (this is the branching step).

A list is used to store the boxes and the bounds of on each box. Since a right bound of

on a box is an upper bound for the global minimum, the least right bound can be used

´ µ to discard boxes. In fact, any box for which the left bound of is greater than cannot contain any point of the minimizer and, thus, can be discarded (this is the bounding step). At any time, the interval with the least left bound contains the global minimum, and

the union of the boxes remaining in the list encloses the minimizer. Several algorithms [53, 30, 22, 51, 23] have been developed based on this core idea. The algorithm in [30], due to Ichida and Fujii, improves on the algorithm due to Skel- boe [53] by using a value of the objective function inside a box to get a tighter upper bound for the global minimum. The two algorithms order the boxes in the list using the left bound

of on each box. The union of the boxes returned by both algorithms is in general too large: the boxes kept in the list may have subboxes that can be discarded. Hansen [22] proposed an algorithm that improves on both algorithms by using a differ- ent ordering of the boxes. More specifically, the boxes are ordered either by their widths or by their ages in the list. This allows the algorithm to discard every box that need to be discarded and to converge to all the points in the minimizer [51]. A common feature of the three algorithms is that in the bounding step a box is either discarded or kept as a whole. Hansen and Sengupta suggested a further improvement that allows the remaining boxes to be reduced [23]. This gives rise to the branch and narrow technique. This technique prunes not only by eliminating the boxes for which the left 3.1 Motivation 23

bound of is greater than the upper bound for the global minimum but also by reducing

the boxes not pruned. Hansen and Sengupta use the inequality , where is the best known upper bound for the global minimum, to narrow the boxes not discarded by branch and bound.

The four algorithms, however, have the following drawbacks, which limit their capabili- ties:

They all mix the fathoming problem and the localization problem. In fact, for some

optimization problems we may know the global minimum and we only need to deal

´µ  ¼

with the localization problem. For instance, an equation of the form can be

¾

´µ  ´µ viewed as a global optimization problem with an objective function , and for which the global minimum is equal to 0. For this particular example, the only concern is the localization problem and not the fathoming problem.

They need to be extended to include outer approximation and inner approximation for -minimizer. In several cases the -minimizer is more interesting than the minimizer.

In these cases, we need to find boxes that are contained within the -minimizer (inner approximation), and boxes that cover it (outer approximation). This is found to be a useful tool for sensitivity analysis and optimal engineering system design.

They do not say whether their results are the best results that one can compute using

interval arithmetic, a fixed expression of , and a specified arithmetic precision. In fact, the properties of the four algorithms are stated only in the limit for infinite run- ning time, infinite memory, and infinite precision of the floating-point number system (see for example Ratschek and Rokne [51]). In this chapter we find properties that can be verified in actual executions of the algorithms mentioned above. Moreover, as we shall show later, our proposed improvements return the best results that can be

computed with a given expression for and a given hardware.

These drawbacks can be eliminated by incorporating the following improvements [62]:

Separating the fathoming problem and the localization problem.

3.2 The fathoming problem 24 Computing inner and outer approximations for the -minimizer.

Both of these are discussed in detail in the next sections.

3.2 The fathoming problem

The fathoming problem can be solved by answering the following two questions:

First Fathoming problem: what is the best possible lower bound for Ñ
Ò ?

Second Fathoming problem: what is the best possible interval for Ñ
Ò ? By the “best possible” lower bound or interval we mean, respectively, the lower bound or

the interval that cannot be improved using interval arithmetic given an expression for and an arithmetic precision.

The second fathoming problem is more general than the first fathoming problem: the

Ñ
Ò left bound of the best possible interval for Ñ
Ò is the best possible lower bound for . In his essay “Proofs and Refutations” [35], Lakatos presented a rational reconstruction of the proof of Euler’s conjecture about polyhedra. He gave, based on an imaginary di- alogue between a teacher and his students, the evolution of an initial idea that stemmed from Cauchy to the final proof of the conjecture. Lakatos, in addition, presented the actual history of the evolution of the proof as footnotes. Using the same idea, we shall present a

rational reconstruction of an algorithm that solves the second fathoming problem.



Æ ½

In what follows, we assume that we have a cover ¼ for , and that

Æ ½

the cover is ordered by non-decreasing left bounds of over the boxes ¼ .

  ½

Initially, we consider ¼ and . Such a cover contains the minimizer. We shall consider algorithms that change a given cover containing the minimizer to one that has a smaller union and still contains the minimizer.

3.2 The fathoming problem 25



let ¼ ;

let the cover be ¼ ;

´ µ

while ( atomic ¼ )

´ µ ¼

// invariant: Ñ
Ò and is a cover for (thus, it is a cover for the minimizer)

remove ¼ from the cover;

split ¼ and insert the results of splitting into the cover in non-decreasing order according

to the left bound of ;

´ µ ¼

// Ñ
Ò

´ ´ µµ

output lb ¼ ; ´µ

Figure 3.1. The algorithm MS ¼ . The function atomic specifies whether its box argument is atomic or not.

3.2.1 The first fathoming problem

1

In [53], Skelboe presented an algorithm that computes a lower bound for Ñ
Ò . Essentially,

the algorithm is as shown in Figure 3.1.

The algorithm MS¼ stores the boxes to be processed in a cover for . The cover is

sorted using the left bound of over the boxes (Skelboe’s heuristic). The algorithm MS¼

splits the first box ¼ in the sorted cover until it becomes atomic.

½ The algorithm presented by Skelboe appeared on 1974, and was intended for computing tighter bounds for the range of a rational function. His algorithm first searched for a lower bound and then an upper bound

for the range of over a box. The termination criterion of the algorithm was that the relative error between

the current and the previous lower bounds has to be less than a predetermined ¯. This algorithm was then improved by Moore on 1976 [45]. Moore mainly used Skelboe’s heuristic of ordering the boxes in the list as well as monotonicity tests, the centered form, and Krawczyk’s version of Newton’s method. Today, the original Skelboe algorithm is often referred to as the Moore-Skelboe algorithm. Note that the phrase “Global Optimization” was not used in [53] and [45]. In the context of interval arithmetic, the phrase “Global Opti- mization” first appeared in Hansen’s paper [21], in Skelboe’s paper [54], and in Ichida and Fuji paper [30]. 3.2 The fathoming problem 26

The properties of this algorithm are stated in Theorem 1.

¼

Theorem 1 Algorithm MS terminates and outputs the best possible lower bound for Ñ
Ò .

Proof. At each step of the algorithm ¼ , the list is a cover for . This implies that

´ µ

Ñ
Ò has to be in for a certain box in the cover . Since, the boxes in the cover

´ µ ¼

are ordered in a non-decreasing order of the left bounds of , Ñ
Ò has to be in ,

¼

where is the first box in the cover . The termination of the algorithm MS¼ results from

´ ´ µµ ¼

being finite, and it outputs the best lower bound because ¼ is atomic, lb is a

´ ´ µµ

Ñ
Ò ¼

lower bound for Ñ
Ò , and can be equal to lb .

¼ In addition to returning the best lower bound for Ñ
Ò , the algorithm MS subdivides the boxes in an adaptive way, as explained below.

Let us consider the operation “split” in the algorithms MS¼ . It assumes that the results are nonempty, are both proper subsets, and have a union that is equal to the box that was

split. The box to be split is a Cartesian product of intervals. So the split can be done in

up to different ways. The number of boxes in the cover created by the algorithm typically becomes so large that the cover cannot be stored. So one should be careful what to split.

It is desirable to split a box most likely to contain the minimizer. The heuristic chosen by

´ µ

¼ ¼ the algorithm MS is to split the box ¼ for which has the least left bound. The subdivision resulting from the splits in this algorithm is adaptive: boxes far away from the global minimum tend not to be split. This should go some way towards avoiding covers with too many boxes.

Although the algorithm MS¼ has an efficient heuristic for choosing the box to split, it

only partially solves the fathoming problem2. In practice, it is often the case that an interval containing Ñ
Ò is desired. As we shall see, by modifying of the termination criterion and

¾ The Moore-Skelboe algorithm is an example of algorithms that solve only the first fathoming problem.

These kind of algorithms, however, are not sufficient for global optimization; returning a lower bound does

not give that much information about the location of Ñ
Ò .

3.2 The fathoming problem 27



let ¼ ;

let the cover be ¼ ;

´ ´ µµ

while ( ¼ )

´ µ ¼

// invariant: Ñ
Ò and is a cover for (thus, it is a cover for the minimizer)

remove ¼ from the cover;

split ¼ and insert the results of splitting into the cover in non-decreasing order according

to the left bound of ;

´ µ ´ ´ µµ

¼ ¼

// Ñ
Ò and

´ µ

output ¼ ;

Figure 3.2. The algorithm MS ½ . The function has as value the width of its interval ar-

gument. The tolerance is assumed to be sufficiently large as termination of the algorithm is not assured otherwise.

the output of the algorithm MS¼ , we obtain an algorithm that solves the second fathoming problem.

3.2.2 The second fathoming problem

Instead of directly computing the best possible interval for Ñ
Ò , we deal with finding an

¼ interval for Ñ
Ò of width at most , for some real . As we shall see later, the best

possible interval can be obtained by setting to zero.

´ µ

Ñ
Ò ¼

From the algorithm MS¼ , we know that is in . A reasonable criterion for

´ µ ½ termination is to stop when the width of ¼ is less than . See the algorithm MS shown in Figure 3.2.

The algorithm MS½ has some positive features. In the first place, it may find a sufficiently

narrow interval for Ñ
Ò . Second, it does this by subdividing in an adaptive way as in the case of the algorithm ¼ . 3.2 The fathoming problem 28

However, the algorithm MS½ needs improvement in several ways. For example, what

happens if one chooses a too small positive ? Even if the algorithm does not abort because the number of boxes in the cover has become too large to be stored, ¼ may become the result of so many splits that all its projections are atomic intervals. The split operation should be such that both of the resulting boxes are smaller than the original, and this is then no longer possible. So let us assume a boolean function “atomic” that tells whether its box argument can be split into two boxes that are both smaller. Let us also address the problem of the cover becoming too large in both algorithms

3

´ ´ µµ

½


¼ and . Suppose there are boxes and in the cover such that lb

´ ´ µµ
rb  , then does not intersect with the minimizer and we can remove it from the

cover while maintaining the order of the cover. Therefore, it is of interest to maintain in the

algorithm an upper bound for the global minimum. An example of such a is the least of the right bounds of on the boxes in the cover. A tighter upper bound can be obtained

4 5 by using the right bound of at a point inside a box . We thus obtain the algorithm MS¾ shown in Figure 3.3

The algorithm MS¾ is an improvement because it has a defined outcome for every non-

negative value of . Moreover, the “best success” outcome is useful because it produces the best possible lower bound for Ñ
Ò . This enables us to formulate Theorem 2.

¿ Skelboe in his Algol program [53] checks whether the cover is too large to be stored. If this is the case, he outputs “STORAGE LIST OVERFLOW”. As far as we know, the first paper that addresses the capacity problem of the cover was Ichida and Fujii paper [30]. Ichida-Fujii algorithm is thus the first algorithm to use

branch-and-bound technique and interval arithmetic to do global optimization.



The use of midpoints as a way to obtain upper bounds for Ñ
Ò first appeared in Ichida and Fujii pa- per [30]. Since then, the midpoint test is, to our knowledge, copied uncritically by all global optimization

scholars. 

The algorithm MS¾ is similar to the one given by Ichida and Fujii [30]. More precisely, Ichida-Fujii

  ¢  ¢ 

Ñ ½

algorithm uses only one criterion for the termination; given a box ¼ , the width of the

  

Ñ ½ largest interval in ¼ has to be smaller than a predetermined precision. The first paper that had both criteria was Hansen’s paper [21].

3.2 The fathoming problem 29

let ¼ = ;

let the cover be ¼ ;

 ´ ´ µµ

¼ ¼

let ¼ be a point of and let rb ;

´ ´ µµ ´ µ ¼

while ( lb ¼ and atomic )

 ´ ´ µµ  ¼

// invariant: Ñ
Ò lb and is a cover for the minimizer

remove ¼ from the cover;

split ¼ and insert the results of splitting into the cover in non-decreasing

order according to the left bound of ; update using a point of the box with the least right bound of on the boxes in the cover;

remove all boxes from the cover with left bound greater than ;

´ µ ´ µ ¼

// ¼ or atomic

´ ´ µµ

if ( lb ¼ )

 ´ ´ µµ  ¼ output “success: Ñ
Ò lb ”;

else

 ´ ´ µµ  ´ ´ µµ

¼ ¼

output “best success: Ñ
Ò lb , and lb

is the best possible lower bound for Ñ
Ò ”;

Figure 3.3. The algorithm MS ¾ . In case of success, an interval of the desired width has been found. Otherwise, a lower bound for the global minimum has been found that is the

best possible one, given the expression for and the precision of the arithmetic. 3.2 The fathoming problem 30

Theorem 2 Algorithm MS¾ terminates and, in case of “best success”, it returns the best

lower bound.

Proof. Since the cover is a cover for the minimizer, Ñ
Ò has to be in one of the

´ ´ µµ

boxes in the cover. Because the boxes are ordered by non-decreasing order of lb

 ½ ½

for , and is an upper bound of at a point in , Ñ
Ò has to be in

 ´ ´ µµ  ½ lb ¼ . The termination of MS is implied by the fact that the number of floating- point numbers is finite. Therefore the total number of the boxes that can be defined is finite. Every split changes a non-atomic box into two strictly smaller boxes. A non-terminating

loop would therefore generate an infinite sequence of different boxes. Termination of the

 ´ ´ µµ ´ µ

¼ ¼

algorithm follows. It is possible that Ñ
Ò lb . The interval is therefore the

´ ´ µµ ¼

only interval known to contain Ñ
Ò . The only way to improve lb as lower bound

¼ ¼

for Ñ
Ò is to bisect . Once is atomic, this lower bound cannot be further improved.

¾

Some experimental results related to MS¾ are given in Section of Chapter .

A question arises: does it also produce the best possible interval for Ñ
Ò ? The answer

¼

is that this is not necessarily the case. Algorithm MS¾ only splits . The interval also

depends on the upper bound for Ñ
Ò . It may be that splitting boxes of the cover other

than ¼ results in a lower . The heuristic for selecting the box to split must be well-

chosen. In fact, if the upper bound of the box selected is far away from , then the chance

to improve is extremely small. A practical heuristic is to choose a splittable box, say

´ µ

for which
has the least right bound. The main limitation of the algorithms above is the large number of boxes in the cover.

Removing the boxes of the cover whose left bound exceeds is essential to get better

time-efficiency. One way to get better time-efficiency is to use as a source for the values

of on a point inside the boxes of the cover. These are also upper bounds for the global minimum and are less than the upper bound obtained by interval arithmetic. Instead of picking any point somewhere in the box, several authors use the midpoint (the center) of 3.2 The fathoming problem 31

the box because, as it may seem, a midpoint would give a better upper bound for the global minimum. However, this is unwarranted because a midpoint may not provide any more information than other points. Hence, we do not need to spend extra processing time to

compute the midpoint of a box; instead we can use a point that is readily available such as

one of the corners. This is what we do in our final algorithm ¿ . This algorithm is given

in Figure 3.4.6

As pointed out earlier, the best lower bound for Ñ
Ò is reached when a certain box,

Ñ
Ò namely ¼ , is atomic. This condition is not sufficient to get the best upper bound for .

Although the boxes remaining in the cover may all be atomic, we may get a better upper bound by evaluating at the corners of the remaining boxes.

The power of the algorithm MS¿ is stated in Theorem 3.

Theorem 3 Algorithm MS ¿ terminates and, in case where is zero, outputs the best inter-

val for Ñ
Ò .

Proof. The proof for this theorem is the same as for Theorem 2 except that we must prove

that when is sufficiently small, if necessary equal to zero, the interval returned by the algorithm has the best upper bound for Ñ
Ò .

By choosing sufficiently close to zero, we force the boxes to be atomic or to be re-

moved from the cover. The corners of the remaining boxes are used to get the best upper

 ´ ´ µµ 

¼ Ñ
Ò bound for Ñ
Ò . As a result, the interval lb is the best interval for that

one can compute with the given expression of and the given precision of the arithmetic.

 ÅË

The algorithm ¿ is similar to the “interval-valued fathoming algorithm” [62] except that we check

the corners to obtain the best upper bound. The “interval-valued fathoming algorithm” was claimed to obtain

the best interval containing Ñ
Ò . This, however, is not always true. Although the boxes remaining in the

cover after the termination of the “interval-valued fathoming algorithm” are all atomic, we may still get better

upper bound for Ñ
Ò by using one of the corners of these boxes. In other words, all the corners of the boxes

in the cover must be checked in order to obtain the best interval for Ñ
Ò .

3.2 The fathoming problem 32

let ¼ = ;

let the cover be ¼ ;

let be the upper bound of at one of the corners of ¼ ;

´ ´ µµ

//make the lb ¼ as high as possible

´ ´ µµ ´ µ ¼

while ( lb ¼ and atomic )

 ´ ´ µµ  ¼

// invariant: Ñ
Ò lb and is a cover for the minimizer

remove ¼ from the cover;

split ¼ and insert the results of splitting to the cover in non-decreasing

order according to the left bound of ;

from the results of splitting, choose the box with the smallest right bound of ;

update using the upper bound of at one of the corners of ;

remove all boxes from the cover with left bound greater than ;

´ ´ µµ ´ µ ¼ // lb ¼ or atomic

//make as low as possible

´ ´ µµ ´ µ

while ( lb ¼ and such that atomic )

 ´ ´ µµ  ¼

// invariant: Ñ
Ò lb and is a cover for the minimizer

remove
with least right bound from the cover;

split
and insert the results of splitting into the cover in non-decreasing

order according to the left bound of ;

from the results of splitting, choose the box with the smallest right bound of ;

update using the upper bound of at one of the corners of ;

remove all boxes from the cover with left bound greater than ;

(Continued next.) 3.2 The fathoming problem 33

//check all the corners of the boxes left in the cover

´ ´ µµ

if ( lb ¼ ) //all the boxes left in the cover are atomic

//we may get a better results by evaluating at the corners of the boxes

for (every box in the cover)

update by evaluating at the corners of ;

remove all boxes from the cover with left bound greater than ;

´ ´ µµ

if ( lb ¼ ) exit from the for loop;

´ ´ µµ ¼ ½ ´ µ

// ¼ or atomic

´ ´ µµ

if ( lb ¼ )

 ´ ´ µµ  ¼ output “success: Ñ
Ò lb ”;

else

 ´ ´ µµ  Ñ
Ò output “best success: lb ¼ is the best interval for ”;

Figure 3.4. The algorithm MS ¿ . In case of success, an interval of the desired width has been found. Otherwise, an interval for the global minimum has been found that is best,

given the expression for and the precision of the arithmetic.

3.3 The localization problem 34

¾

Some experimental results related to MS¿ are given in Section of Chapter .

We call special attention to the fact that the algorithm MS¿ is the best one can do as long

as the expression of and the precision of the arithmetic are fixed. Changing, for example,

the expression of may lead to a different outcome.

3.3 The localization problem

The localization problem is to gain information about the Æ -minimizer. This can take two forms: an outer approximation or an inner approximation.

3.3.1 The outer approximation

An outer approximation is a set of boxes whose union contains the Æ -minimizer. This can always be achieved, though the union may be so large as not to be useful. The algorithm in Figure 3.5 makes this union smaller.

An important property of algorithm MS is given in Theorem 4. Æ

Theorem 4 Algorithm MS  terminates and gives the best outer approximation to the -

minimizer.

Æ ´µ · Æ ´µ

Proof. Let be an element of the -minimizer. Then Ñ
Ò and hence

· Æ

. This implies that there is a box in the cover containing because only boxes with

´ µµ · Æ

lb´ have been removed from the cover. Because is the best upper bound

Ñ
Ò

for and because the boxes left in the cover are all atomic, the algorithm MS outputs the best possible approximation of the Æ -minimizer for the given expression of and the

given arithmetic precision.

¾

Some experimental results related to MS are given in Section of Chapter .

Figure 3.6 illustrates the outer approximation of the Æ -minimizer. It is worth noting that the boxes in the algorithm MS are ordered using the left bound of on each box.

3.3 The localization problem 35

let ¼ = ;

let the cover be ¼ ;

let be the right bound of at one of the corners of ¼ ;

´ µ while ( such that atomic  )

//the cover is an outer approximation for the Æ -minimizer

´ ´ µµ

choose a nonatomic box
with the least right bound rb ;

remove
from the cover;

split
and insert the results into the cover in non-decreasing

order according to the left bound of ;

from the results of splitting, choose the box with the smallest right bound of ;

update using the right bound of at one of the corners of ;

· Æ

remove all boxes from the cover with left bound greater than ;

//all the boxes left in the cover are atomic

for (every box in the cover)

update by evaluating at the corners of ;

· Æ

remove all boxes from the cover with left bound greater than ;

Figure 3.5. The algorithm MS  . After termination, the best outer approximation to the

Æ -minimizer is the set of boxes left in the cover. 3.3 The localization problem 36

Outer approximation boxes f values Discarded

Delta U

Epsilon

Lb(f(Bo)) Æ

Figure 3.6. A terminal situation of the algorithm MS  outer approximation for the - minimizer. 3.3 The localization problem 37

3.3.2 The inner approximation Æ An inner approximation is a set of boxes that are subsets of the Æ -minimizer. If is too

small, then we can only obtain an empty inner approximation. For a sufficiently large Æ ,

however, this inner approximation can be made larger as explained below.

Let and be the right and the left bound of the best interval for Ñ
Ò . These bounds



are computed using the algorithm MS¿ . We consider , which is the width of the

´ ´ µµ · Æ

best interval for Ñ
Ò . Let be a box such that rb . Such a box will

· Æ µ Æ

now be contained in the ´ -minimizer for any positive . Hence, the boxes of the cover

· Æ ´ · Æ µ that have right bounds less than are in the inner approximation of -minimizer. This inner approximation can be improved by splitting certain boxes. This improvement is carried out by the algorithm in Figure 3.7.

The relevant characteristics of algorithm MS have been incorporated into Theorem 5

·

Theorem 5 Algorithm MS terminates and gives the best inner approximation to the (

Æ )-minimizer.

´µ ´ µ

Proof. Suppose for one of the boxes in the cover. Then and

¼ ¼

´µ ´ ´ µµ ´ ´ µµ · Æ

hence rb . For all boxes in the cover, we have rb .As

· ´µ · Æ · ´ · Æ µ

Ñ
Ò Ñ
Ò

,wehave .Soitisinthe -minimizer. Algorithm MS

¼ ¼

· Æ ´ µ returns the best inner approximation because the the boxes for which are all atomic and their corners are all processed to determine whether they are in the inner

approximation or not.

¾

Some experimental results related to MS are given in Section of Chapter .

· Æ µ Figure 3.8 illustrates the inner approximation of the ´ -minimizer. In the case of the inner approximation, note that the boxes in the cover are ordered using

the right bound of on each box. This is to be contrasted with the order used in the outer approximation.

3.3 The localization problem 38

let ¼ = ;

let and be the bounds obtained from MS ¿ ;



let ;

let the cover be ¼ ;

´ · Æ µ ´ µ ´ µ

while( such that
and atomic )

´ · Æ µ ´ µ

//the set of the boxes in the cover with

· Æ µ

//is an inner approximation to ´ -minimizer

remove
from the cover;

split
and insert the results into the cover in non-decreasing

order according to the right bound of ;

· Æ

remove all boxes from the cover with left bound greater than ;

´ µ ´ ´ µµ ´ · Æ µ
// atomic
or rb

//check the corners of the boxes in to see if they are in the inner approximation,

´ · Æ µ ´ ´ µµ

//and remove boxes with rb from

´ · Æ µ ´ ´ µµ

for (every non-singleton box in the cover such that rb )

´ ´µµ · Æ

if (a corner of satisfies rb )

add to the cover;

remove from the cover;

Figure 3.7. The algorithm MS  . After termination, the set of boxes left in the cover is the

· Æ µ best inner approximation for the ´ -minimizer. 3.3 The localization problem 39

f values Discarded

Inner approximation

Delta U

Epsilon

Lb(f(Bo))

´ ·

Figure 3.8. A terminal situation of the algorithm MS  for inner approximation of the µ Æ -minimizer. Chapter 4

Constrained global optimization via interval arithmetic

From a numerical point of view, it would be much easier if all global optimization problems were unconstrained. Unfortunately, this is not so. It is often the case that a mathematical model of a system involves conditions on the values of the parameters of the system. Even worse, for several cases such conditions are nonlinear. This leads to the most general global optimization problem, that is, the constrained global optimization problem. In this chapter, we shall deal with constrained global optimization problems from the perspective of interval arithmetic. Although the algorithms presented are minor modi- fications of existing algorithms, they are, to our knowledge, the first constrained global optimization algorithms with provable performance. In the first section of this chapter, we shall present an overview of the interval arithmetic methods for solving constrained global optimization problems. Because of the presence of conditions in a constrained global optimization problem, we need not only to solve the fathoming problem and the localization problem, but also to compute the set of feasible points. Due to machine limitations, instead of computing this set we compute its best inner approximation and its best our approximation as done in the second section. These approx- imations are used, in the last section, to solve the fathoming problem and the localization problem. 4.1 Overview 41

Throughout this chapter, we assume that is specified by System (1.2).

4.1 Overview

The complexity of constrained global optimization over unconstrained global optimization is due mainly to proving that a box contains feasible points. More precisely, Papadimitriou and Steiglitz [50] showed that solving system of equations and solving constrained global optimization have the same complexity. A feasible point is a point that satisfies all the conditions in a constrained global op-

timization. As opposed to unconstrained global optimization, the right bound of on a box is not necessarily an upper bound for the global minimum unless has at least one

feasible point. To prove that a box contains feasible points, we either find, by testing, a

feasible point in the box , or make use of Moore’s existence theorem [23, 46]. A better way is to make use of SIVIA, which we explain below.

The first interval arithmetic algorithm that solves the constrained global optimization problem with inequality conditions is due to Hansen and Sengupta [24, 23]. They used,

in addition to branch and bound, several tests to check whether a point in a given box is

feasible. Given a feasible point , the right bound of at is an upper bound for the global minimum. SIVIA (Set Inversion Via Interval Arithmetic) is a recursive interval arithmetic algo- rithm, due to Jaulin et al. [32], that computes an inner approximation and an outer approx-

imation of the set of feasible points, namely the set given by System (1.2). The inner

approximation can be used to update the upper bound for Ñ
Ò , and the outer approximation is used to compute the lower bound for Ñ
Ò . A version of SIVIA is shown in Figure 4.1.

SIVIA, however, may return an empty inner approximation as stated in Theorem 6

Theorem 6 If System (1.2) contains at least one equality that requires at least one opera-

tion with a rounding error, then its inner approximation computed using SIVIA is empty.

´ µ  ¼ Ñ Proof. Assume that we have an equality condition of the form ¼ that requires at least one operation with a rounding error. To encompass the rounding errors,

4.1 Overview 42

initialize Áfs to ; initialize Ç fs to ;

//The function SIVIA is recursive, and can be called

Ò

Á Ç

//as SIVIA(Ê , fs, fs).

//The set Áfs is an inner approximation for ,

//and the set Ç fs is an outer approximation for

Á Ç

function SIVIA(Box , Set fs, Set fs)

´ ´ µµ ¼ Á Ç

if ( rb
) add to fs; add to fs; continue;

´ ´ µµ ¼

if ( lb
) return;

µ

if (atomic ´ ) Ç

add to fs; return;

// is not atomic, so it can be split

Ö

split into Ð and ;

Á Ç Á Ç Ö

SIVIA( Ð , fs, fs); SIVIA( , fs, fs);

´µ Figure 4.1. The algorithm . The function atomic specifies whether its box argu- ment is atomic or not. It finds an inner and an outer approximation for System (1.2). 4.2 The feasibility problem 43

outward rounding has to be used in the interval arithmetic evaluation of . This outward

´ µ

rounding leads to a non-degenerate interval. That is, is a non-singleton interval for

¼ ¼

´ µ  ¼ ¼ any box . This implies that we cannot find any box that satisfies . Thus, the theorem is proved.

Theorem 6 implies that for constrained global optimization problems involving equality conditions, it is difficult to find a feasible point using SIVIA. To overcome this difficulty, Hansen and Walster [19, 23, 51] used Moore’s existence theorem to check whether a box

has feasible points. As a consequence, the right bound of over a box that is proven to contain feasible points is an upper bound for the global minimum. Because Moore’s existence theorem involves derivatives, we will not discuss it any further (since we are dealing with functions that may not have derivatives). In what follows, the problem of finding feasible points is referred to as the feasibility problem.

In the next sections, we shall deal separately with the feasibility problem, the fathoming problem, and the localization problem.

4.2 The feasibility problem

Computing the exact set of feasible points, which is the set , is generally not realizable due to machine limitations. The only thing one can hope for is to find an inner approx- imation and an outer approximation for such a set. As we mentioned earlier, the inner approximation, when it is not empty, is used to update the upper bound for the global mini-

mum, and the outer approximation is used to find the lower bound for the global minimum. Á If the union of the boxes in the inner approximation for is denoted by fs and the union

of the boxes in its outer approximation is denoted by Ç fs, then we obviously have

Ç Áfs fs

Before presenting an improved version of SIVIA we shall separate between the inner

4.2 The feasibility problem 44

initialize Ç fs to ;

Ò

Ê

initialize the active set  to ;



while ( ) 

remove from the active set ;

´ ´ µµ ¼

if ( lb
) continue;

´ ´ µµ ¼ ´ µ Ç

if ( rb
atomic ) add to fs; continue; 

split and insert the results of splitting into the active set ;

//The set of the boxes in Ç fs is an outer approximation of

 Ç

if (Ç fs ) output the boxes in fs;



else output the empty list ; // ´µ

Figure 4.2. The algorithm FS Ç . The function atomic specifies whether its box argument is atomic or not.

approximation, and the outer approximation for . This separation is just for clarity pur-

Ç Á poses. The two algorithms, FS Ç and FS , presented below, make respectively fs as small as possible, and Á fs as large as possible. 4.2.1 The outer approximation

An outer approximation for the feasible region can be computed using the algorithm FSÇ

shown in Figure 4.2.

 Ç The algorithm FSÇ uses two lists and fs. The first list stores the boxes that still need

to be processed, and the second list contains the boxes that are in the outer approximation.

The correctness of the algorithm FSÇ is based on the fact that a box can only be in

one of the following cases:

´ ´ µµ ¼

1. There exists an such that lb
. In this case, the box does not satisfy all the conditions in the constrained global optimization problem; hence, it must be discarded.

4.2 The feasibility problem 45

´ ´ µµ ¼

2. For every ,wehaverb
. This means that the box satisfies all the conditions in the constrained global optimization problem. It must be therefore added

as whole to the outer approximation list Ç fs.

´ ´ µµ ¼ ´ ´ µµ ¼

3. None of the above, that is for every , lb
and rb . The box Ç

in this case is in an “undetermined” state; may or may not be an element of fs.If

Ç is atomic, then is inserted into fs. Otherwise, it is split and added to the active

set  for further processing.

The main feature of the algorithm FSÇ is to make the outer approximation for as small as possible. This feature is stated in Theorem 7.

Theorem 7 Algorithm FS Ç terminates and outputs the best outer approximation for the

feasible set .

The termination of the algorithm FSÇ follows from the fact that the number of floating- numbers is finite. The algorithm outputs the best outer approximation because the boxes that are in the undetermined state are all atomic. Thus, we cannot discard them. Therefore, the outer approximation cannot be made smaller.

4.2.2 The inner approximation

An inner approximation for the feasible set is a set of boxes whose points satisfy all the conditions in the constrained global optimization problem. The set containing the empty

set is an inner approximation for , but it is a useless inner approximation. The algorithm shown in Figure 4.3, tries to make such an inner approximation as wide as possible.

The distinctive property of the algorithm FSÁ is the following.

Theorem 8 Algorithm FS Á terminates and outputs the best inner approximation for the

feasible set .

The correctness of the algorithm FSÁ is based on the following facts:

´ ´ µµ ¼

A box for which rb
for every satisfies all the conditions in the

4.2 The feasibility problem 46

initialize Áfs to ;

Ò

Ê

initialize the active set  to ;



while ( ) 

remove from the active set ;

´ ´ µµ ¼ Á

if ( rb
) add to fs; continue;

´ ´ µµ ¼

if ( lb
) continue;

µ

if (atomic ´ )

´ ´ µµ ¼

if (any vertex of satisfies rb
)

Á add to fs;

continue;



split and insert the results of splitting into the active set ;

//the set Áfs is an inner approximation of

 if (Áfs )

output the boxes in Áfs; //“success” else

output the empty list; //“best success” ´µ

Figure 4.3. The algorithm FS Á . The function atomic specifies whether its box argument is atomic or not. 4.2 The feasibility problem 47

constrained global optimization problem. Hence, it is added to the list of inner ap-

proximation Áfs.

´ ´ µµ ¼

A box for which lb
for at least one does not satisfy all the conditions

in the constrained global optimization problem. Therefore, it must be discarded.

If none of the above, then we have two cases: either is atomic, or not. If is

not atomic, then it must be split and added to the active set  for further processing.

Otherwise, we must check if any vertex of is feasible. If this is the case, the box

Á

must be added to the inner approximation list fs. Á Before proceeding to the next section, it is worth noting that the algorithms FSÇ and FS can be combined into the algorithm FS shown in Figure 4.4.

Note that in the algorithm FS we used  fs to store the boxes that are in the outer ap-

proximation and not in the inner approximation. This allows us to avoid the duplication of

Ç  Á Ç

boxes in Áfs and fs. The relation between fs, fs, and fs the outer approximation is

Á   Ç fs fs fs

It is also worth pointing out that the boxes in  fs are all atomic. The algorithm FS can easily be implemented so that the boxes are not processed until they are atomic; the algorithm can be forced to treat only boxes with size greater than a predetermined precision. Our algorithm FS improves on SIVIA in the sense that it checks the vertices in order to get the best inner approximation. Note also that the algorithm FS does not use recursion. The reason behind such a choice is that recursion allows only a depth first strategy for

processing boxes. Using the active set instead permits us to choose any heuristic to order

the boxes in .

We call special attention to the fact that the algorithm FS and SIVIA do unnecessary

¼

work; they check any subbox of a box against all conditions even in cases where satisfies some of these conditions. It is obvious that these algorithms need to check the

4.2 The feasibility problem 48

initialize Áfs to ;

initialize  fs to ;

Ò

Ê

initialize the active set  to ;



while ( ) 

remove from the active set ;

´ ´ µµ ¼ Á

if ( rb
) add to fs; continue;

´ ´ µµ ¼

if ( lb
) continue;

µ

if (atomic ´ ) 

add to fs

´ ´ µµ ¼

if (a vertex of satisfies rb
)

Á add to fs;

continue;



split and insert the results of splitting into the active set ;

//the set Áfs is the best inner approximation for ,

 //and the union of Áfs and fs is the best outer approximation for .

Figure 4.4. The algorithm FS. The function atomic ´µ specifies whether its box argument is atomic or not.

4.3 The fathoming problem 49

¼ subbox only against those conditions not satisfied by its parent box . An improved algorithm is shown in Figure 4.5.

To each box we associate a Boolean vector of size equal to the number of conditions

in the constrained global optimization problem. The ith component of corresponds to

´ ´ µµ ¼

whether we know that the box satisfies the condition
.

Ò

 Ê

Initially, the component of the vector associated with ¼ are all set to false.

´ µ ´ µ ´ µ

For each box , if the ith component
of is false, then we must evaluate .

´ µ

Otherwise, we do not need to. Note that when
is true for all , the box must be

added to the inner approximation list. Á

In what follows, the union of the boxes in Áfs is denoted by fs and the union of the Ç boxes in Ç fs is fs. 4.3 The fathoming problem

Suppose that we want to compute an interval of width less than containing the global Ç

minimum. In addition, suppose that the two sets Á fs and fs are both available. We also

 suppose that Ç fs . The algorithm that solves the fathoming problem is given in Fig- ure 4.3.

The algorithm FPFS is similar to the algorithm MS¾ given in the previous chapter except

that:

to update the upper bound for Ñ
Ò using the right bound of on a box ,wehave Á

to ensure that the box intersects with fs. not only are the boxes with left bound greater than , found so far, removed from the

cover but also those that do not intersect with Çfs. The main property of the algorithm FPFS is the following.

Theorem 9 Algorithm FPFS terminates and, in case where is zero, outputs the best in-

terval for Ñ
Ò .

4.3 The fathoming problem 50

Ò

 Ê

¼

initialize Áfs to ;

initialize  fs to ;



initialize the active set to ¼ ;

´ µ

initialize the components of ¼ to false;



while ( ) 

remove from the active set ;

´ µ

if (
)

´ ´ µµ ¼

if (lb
) continue;

´ ´ µµ ¼ ´ µ 

if (rb
) true;

´ µ Á

if (
) add to fs; continue;

µ

if (atomic ´ ) 

add to fs

´ ´ µµ ¼

if (any vertex of satisfies rb
)

Á add to fs;

continue;



split and insert the results of splitting into the active set

´ µ

and set their associated vectors to ;

//the set Áfs is an inner approximation of ,

 //and the union of Áfs and fs is the best outer approximation for .

Figure 4.5. An improved version of the algorithm FS. This version avoids processing boxes for which the parents are proved to satisfy some of the conditions involved in the constrained global optimization problem.

4.3 The fathoming problem 51

Ç

let ¼ be the smallest box enclosing fs;

initialize the cover to ¼ ;

 Á

if (Áfs ) set to the right bound of on a point in fs;

´ ´ µµ

else set to rb ¼ ;

´ ´ µµ

//make the lb ¼ as high as possible

´ ´ µµ ´ µ ¼

while ( lb ¼ and atomic )

 ´ ´ µµ  ¼

// invariant: Ñ
Ò lb

remove ¼ from the cover;

Ç 

if ( ¼ fs ) continue;

split ¼ and insert the results of splitting to the cover in non-decreasing

order according to the left bound of ;

if (one of the results of splitting intersects Áfs) update ;

´ ´ µµ Ç 

remove from the cover any box that satisfies lb ,or fs ;

(Continued next.) 4.3 The fathoming problem 52

//make as low as possible

´ µ ´ µ ¼

// ¼ or atomic

 ´ ´ µµ

lb ¼ ;

remove from the cover all boxes that do not intersect Áfs;

´ µ

while ( and such that atomic
)

 

// invariant: Ñ
Ò

remove a non-atomic
with least right bound from the cover ;

split
and insert the results of splitting into the cover in non-decreasing

order according to the left bound of ;

if (one of the results of splitting intersects Áfs) update ;

´ ´ µµ Ç 

remove from the cover any box that satisfies lb or fs ;

´ µ ¼ ½ ´ µ

// or atomic

µ  

if (( ) output “success”: Ñ
Ò ;

  else output “best success”: is the best interval for Ñ
Ò ;

Figure 4.6. The algorithm FPFS. In case of success, an interval of the desired width has been found. Otherwise, an interval for the global minimum has been found that is best,

given the expression for and the precision of the arithmetic.

4.4 The localization problem 53 Ç An algorithm which does not assume the availability of Áfs and fs is given in Fig- ure 4.7.

The algorithm FPFS½ is similar to the algorithm BBCGO in [9] except that we allow our

algorithm to reach atomic boxes. Note that both algorithm FPFS½ and BBCGO have the

same drawback as the algorithm FS; the subboxes of a box are checked using all con-

ditions even though might satisfy some of them. The same idea used to overcome this

drawback for the algorithm FS applies to the algorithm FPFS½ .

An interesting property of the algorithms FPFS½ over BBCGO and the algorithm given

by Ratschek and Rokne in [51] is that FPFS ½ returns the best possible results. More

precisely, in the algorithm FPFS½ , the vertices of an atomic box that satisfies neither

´ ´ µµ ¼ ´ ´ µµ ¼

rb
, nor lb , are checked against the conditions. It may hap-

´ ´ µµ ¼

pen that one of these, say , satisfies rb
, in which case it must be used to

update the upper bound for Ñ
Ò .

¾

Some experimental results related to FPFS½ are given in Section of Chapter .

4.4 The localization problem

In the case of constrained global optimization, the algorithm that computes the outer ap-

proximation for Æ -minimizer is given in Figure 4.8. We suppose that the outer approxima- tion Ç fs of is not empty.

The algorithm LPOFS has the same structure as MS¿ except that we added extra tests to

´ ´ µµ ¼ check whether a box is to be discarded, this is the case when lb
, and

whether to use it to update the upper bound for the global minimum, this is the case when

´ ´ µµ ¼

rb
. An interesting property of algorithm LPOFS is given in Theorem 10

Theorem 10 Algorithm LPOFS terminates and gives the best outer approximation to the

Æ -minimizer.

4.4 The localization problem 54

Ò

 Ê

let ¼ ;

initialize the cover to ¼ ;

´ ´ µµ ¼ ¼

if ( lb
) output the set of feasible points is empty; return;

 ·

let ;

´ ´ µµ

//make the lb ¼ as high as possible

´ ´ µµ ´ µ ¼

while ( lb ¼ and atomic and the cover is not empty)

 ´ ´ µµ  ¼

// invariant: Ñ
Ò lb

remove ¼ from the cover;

split ¼ and insert the results of splitting to the cover in non-decreasing

order according to the left bound of ;

´ ´ µµ ¼

if (a box resulted form the splitting satisfies lb
)

remove from the cover;

´ ´ µµ ¼

if (a box resulted from the splitting satisfies rb
)

update ;

if (a box resulted from the splitting does not satisfy any of the above and atomic(B))

´ ´ µµ ¼

if (a vertex of satisfies rb
)

update ;

remove all boxes from the cover with left bound greater than ;

(Continued next.) 4.4 The localization problem 55

//make as low as possible

´ ´ µµ ´ µ ¼

// lb ¼ or atomic ;

 ´ ´ µµ

lb ¼ ;

´ µ

while ( and such that
))

 

// invariant: Ñ
Ò

remove
with least right bound from the cover;

´ ´ µµ ¼

if ( lb
) continue;

split
and insert the results of splitting into the cover in non-decreasing

order according to the left bound of ;

´ ´ µµ ¼

if (a box that resulted form the splitting satisfies lb
)

remove from the cover;

´ ´ µµ ¼

if (a box resulted from the splitting satisfies rb
)

update ;

if (a box resulted from the splitting does not satisfy none of the above and atomic(B))

´ ´ µµ ¼

if (a vertex of satisfies rb
) update ;

remove all boxes from the cover with left bound greater than ;

´ µ ¼ ½ ´ µ

// or atomic

µ  

if ( ) output “success”: Ñ
Ò ;

 

else output “best success”: is the best interval for Ñ
Ò ;

Figure 4.7. The algorithm FPFS ½ . In case of success, an interval of the desired width has been found. Otherwise, an interval for the global minimum has been found that is best,

given the expression for and the precision of the arithmetic. So this information can be

forced by setting to zero.

4.4 The localization problem 56

Ç ¼

let ¼ be the smallest box enclosing fs, and be the cover ;

 Á

if (Áfs ) set to the right bound of at a point in fs;

´ ´ µµ

else set to rb ¼ ;

´ µ

while ( such that atomic  ) Æ

//the set of boxes in is an outer approximation to the -minimizer

´ ´ µµ

choose an non-atomic box
with least right bound rb ;

Ç 

if (
fs ) remove from ; continue;

remove
from ;

split
and insert the results into ;

Á  if (a box that resulted from splitting satisfies fs )

update ;

Ç 

if (a box that resulted from splitting satisfies fs )

remove from ;

· Æ

remove all boxes from with left bound greater than ;

//all the boxes left in the cover are atomic

for (every box in )

Á

update by evaluating at all the vertexs of that are in fs;

· Æ

remove all boxes from with left bound greater than ;

Figure 4.8. The algorithm LPOFS. After termination, the set of boxes left in the cover

is the best outer approximation to the Æ -minimizer.

4.4 The localization problem 57

· Æ

Similarly, the algorithm that computes the inner approximation for the ( )-minimizer

  is shown in Figure 4.9. The real number is the width of the interval computed using

the algorithm FPFS, and Æ is a real number strictly positive.

The main characteristic of the algorithm LPIFS is stated in Theorem 11.

Theorem 11 Algorithm LPIFS terminates and gives the best inner approximation to the

· Æ ( )-minimizer.

The algorithm LPIFS is similar to the algorithm MS except a for few extra tests.

4.4 The localization problem 58

Ç

let ¼ be the smallest box enclosing fs;

let the cover be ¼ ;

let and be the bounds obtained from FPFS ½ ;



let ;

´ · Æ µ ´ µ Á ´ µ

while( such that (
fs) and atomic )

´ · Æ µ ´ µ Á

//the set of the boxes in with and fs

· Æ µ

//is an inner approximation to ´ -minimizer

remove
from ;

split
and insert the results into in non-decreasing

order according to the right bound of ;

´ ´ µµ · Æ Á 

remove any box from with rb or fs ;

´ µ ´ ´ µµ ´ · Æ µ Á

// atomic
or (rb and fs)

//check the vertexs to see if they are in the inner approximation

´ · Æ µ ´ ´ µµ

for (every non-singleton box in such that rb )

´ ´µµ · Æ Á

if (any vertex of satisfies rb and fs)

add to ;

remove from ;

Figure 4.9. The algorithm LPIFS. After termination, the set of boxes left in the cover is

· Æ µ the best inner approximation to the ´ -minimizer. Chapter 5

Interval constraints

Interval constraints is a novel approach to numerical analysis. It combines constraint pro- cessing developed in artificial intelligence with interval analysis. Interval constraints can be viewed as a generalization of interval arithmetic. Thus, anything that interval arithmetic can do, interval constraints can do it with the same efficiency or even better in some cases. In this chapter we will review the fundamentals of interval constraints, and show how it can be used to evaluate functions, to solve a single inequality, and to obtain box consistency

discussed in Chapter ¾.

5.1 Constraint satisfaction problems

A constraint satisfaction problem (CSP) consists of a finite set of constraints, each involv-

½

ing a finite set of variables. With each variable
, where is in , we associate a

domain
, which is the set of possible values that the variable can take.

A valuation is a tuple indexed by variables; the component indexed by the variable

is an element of
. Given a CSP, the task is to find the values of the variables for which

all the constraints are satisfied. Formally, a solution is a valuation such that each constraint

is true if every variable
in the constraint is substituted by the component of the valuation

indexed by the variable
. The CSP is inconsistent if no solution exists. If every valuation

 Ò

is a solution, then we say that the CSP is totally consistent on the box ½ .

Ò Since the set of solutions of a CSP with variables ½ is a set of tuples (because 5.1 Constraint satisfaction problems 60

it is a set of valuations), can be viewed as a relation, defined by the CSP, on the Cartesian

Ò product of the domains of ½ . This allows us to compose CSPs, and to give them a hierarchical structure. With each constraint, we associate a domain reduction operator; DRO for short, which may remove some valuations that are not solutions. Any DRO is contracting, monotonic, and idempotent as defined below.

Definition 1 (Contracting, Monotonic, and Idempotent Operators) Let © be a mapping

Ò Ò

from to .

Ò

©´ µ ©´ µ

1. © is monotonic iff implies for all , and in .

Ò

©´ µ

2. © is contracting iff for every in .

Ò

©´©´ µµ  ©´ µ 3. © is idempotent iff for every in .

As reported in [59, 3], applying the DROs in a “fair” order causes the domains to converge to a limit. This limit can be empty if one of the domains becomes empty. When this is the case, the CSP is proved to have no solution in the initial domains. If the limit is not empty, then the Cartesian product of the resulting domains is the fixpoint of the fair sequence. In this case, the CSP may or may not have a solution. The word “fair” used above is formally defined as follows. A sequence of DROs is said to be fair if every DRO occurs an infinite number of times in the sequence. Since only a finite number of domains are representable in a computer, any fair se- quence has a finite initial subsequence that terminates at the limit of the fair sequence. An algorithm with such an initial subsequence is called constraint propagation algorithm.

5.1.1 Constraints

Most constraint programming books [4, 57, 10] define a constraint as a relation that holds between its arguments. Since in first-order logic the same definition is used for “atomic formula”, we will adopt this terminology and reserve “constraint” for a special case. In first-order logic, an atomic formula is a predicate symbol that has terms as arguments. Each term can be a variable, a constant, or a function symbol with terms as arguments. So 5.1 Constraint satisfaction problems 61

in an atomic formula a predicate symbol can only occur as the outermost symbol in the formula. The usual way to write atomic formulas, in first-order logic, is by using prefix notation. But in mathematics the most used notation is infix notation. Even though it is known that these notations are equivalent, it helps to show that Inequality (5.1) can be expressed as an

atomic formula.

¾ ¾

· ¼ (5.1)

Using the well-known technique to transform an infix expression to a prefix expression,

the atomic formula corresponding to Inequality (5.1) is

´·´ ´µ ´´ µ ´ µµµ ¼µ (5.2)

From now on, prefix and infix notations will be used to write atomic formulas. The inequalities of System (5.3) are thus atomic formulas.

A constraint is an atomic formula without function symbols. For example, the atomic

´ µ 

formula is a constraint, where is a ternary relation defined by

for all reals , , and .

´ µ Ò

In general, a constraint has the form ½ , where is an n-ary relation with

½

Ò

½ as variables. For every , a value of the variable is consistent

½
·½ Ò

with respect to and domains ½ if and only if

½
½
½
·½
·½ Ò Ò

½ such that

½ Ò

Ò

Let be an atomic formula with variables ½ , and let a relation be its mean-

 Ò

ing. The atomic formula is totally consistent on the box ½ if and only

if .

ADROof is optimal if every domain is reduced to the smallest domain containing all its consistent values. A constraint is considered primitive if it has an optimal DRO that we

5.1 Constraint satisfaction problems 62

µ

know how to compute without iteration. An example, the constraint sum´ defined

· 

by the relation is a primitive constraint, and has the following DRO:

¼

 ´ ´ µµ

¼

 ´ ´ µµ

¼

 ´ ´ · µµ

where the intervals , and are the old domains of , and respectively, and the

¼ ¼ ¼

intervals , and are their new domains.

µ

The constraint Leq´ defined by the relation has the following DRO:

¼

 ´  ´ µµ

rb

¼

´ µ ·µ  ´ 

lb

Before applying propagation algorithm, a CSP must be decomposed into primitive con- straints. As an example, Equation (5.2) is translated into a CSP with the set of primitive

constraints

´ ¼µ ´ µ ´ µ ´ µ ´ µ ´ µ

½ ¾ ¿ ½ ¾  ¿   

5.1.2 Constraint propagation

The Generic Propagation Algorithm (GPA in the sequel) results in an initial subsequence of a fair sequence of DROs. The domains at the end of this initial subsequence are the same as the limit of the entire fair sequence. As all fair sequences have the same limit, GPA computes the unique limit associated with the given CSP and the given initial domains. In general, a GPA maintains a pool of DROs, called the active set, that still need to be applied. These DROs are used to remove inconsistent values as far as possible with modest computation effort. By doing so, we prune the search space, and, thus, gain more information about the solution set. Note that no order is specified for applying the DROs in the active set. Even though many variants of GPA exist (see Apt [3] and Bartak [6]), they are all similar to the pseudo-code given in Figure 5.1. 5.2 Interval constraints and interval constraint solver 63

put the constraints into the active set ;



while ( ) choose a constraint from ;

apply the DRO associated with ; if (one of the domains becomes empty) stop;

add to all constraints involving variables whose domains have changed, if any;

remove from ;

Figure 5.1. A pseudo-code for GPA.

5.2 Interval constraints and interval constraint solver

Any CSP in which all the domains are intervals and all the constraints are primitive is classified under interval constraints. A CSP with the above properties will be referred to as an interval constraint system or an interval constraint satisfaction problem (ICSP). Two CSPs are said to be independent if they have no shared variables.

As we pointed out previously, applying a propagation algorithm to a CSP may cause

a domain to be empty. In this case, has no solutions. A propagation algorithm may also

terminate with small domains for all variables. This, however, is rare. More likely, the

propagation algorithm returns a domain
of a variable that is too wide. In order to

get more information about possibly existing solutions, splitting is then necessary. After

¾

splitting, two CSPs ½ and , that are the same as except for the domain of , are

¾
generated. In ½ , ’s domain is the left half of ;in , it is the right half of . The process of applying propagation and splitting is recursively applied on each CSP that is not proved inconsistent until the domains are small enough. The union of the solutions of the

last CSPs generated, contains the solution of .

5.2 Interval constraints and interval constraint solver 64

´ µ ¼

½ ½ ¾ Ò

´ µ ¼

¾ ½ ¾ Ò

......

´ µ ¼

½ ¾ Ò

´ µ

½ ½¾ ½Ò

allEq ½ ½

......

µ ´

½ Ô¾ ÔÒ

allEq Ô Ô

Figure 5.2. A system of non-linear inequalities without multiple occurrences of variables.

Ò ½ Ô 

Instead, the set ½ is partitioned into equivalence classes where is

½ Ð ÐÕ

½ Ò ½ Ò

a subset  of , for .An constraint asserts that its arguments are equal.

Let us consider a system of inequalities as in System (5.3).

´ µ ¼

½ ½ ¾ Ñ

´ µ ¼

½ ¾ Ñ ¾ (5.3)

......

´ µ ¼

½ ¾ Ñ Since there are many advantages in eliminating multiple occurrences of variables, we rewrite Equation (5.3) to the canonical form shown in Figure 5.2 by introducing new vari-

ables, and the constraint allEq . The variables in Figure 5.2 are grouped into equivalence classes that are associated with each of the variables  in Equation (5.3). All the variables of the same class are equal. Thus, we introduce the predicate allEq . This predicate is true if and only if all its real-valued arguments are equal. It qualifies as a primitive constraint be- cause it has a DRO that is efficiently computed: it replaces the upper bounds of the domains of the arguments by their minimum, the lower bounds by their maximum, and returns fail- ure if the latter exceeds the former. The predicate allEq thus allows us to take into account

information from all inequalities at one time. Another advantage of our translation is that evaluation in interval arithmetic of each expression gives the best possible result, namely 5.2 Interval constraints and interval constraint solver 65

the range of the values of the function associated with the expression. In the next two paragraphs and without loss of generality, we will only consider expres- sions without multiple occurrences of variables.

The tree form of a formula An atomic formula can be represented by a tree in which the predicate symbol is the root, and the arguments, which are terms, are the subtrees of the root. The subtrees related to the terms are defined recursively as follows:

If the term is a variable, then the subtree has only a root, which is that variable.

If the term is a function symbol with one or more arguments, which are terms, then the function symbol is the root of the subtree with the argument terms as subtrees.

Given a tree form of an atomic formula, we label every node that is a function symbol with a unique variable, and we replace any constant that may occur in the formula by a unique variable that has the smallest floating-point interval containing the constant as domain.

Translating a formula to an ICSP The tree form just described can be easily translated

to an ICSP as follows:

´ µ

Ò ½ ¼ Ò ½ The root translates to ¼ where is the predicate symbol and

are the variables labeling the children of the root.

´ µ ´ µ

Ò ½ ¼ Ò ½ A non-root internal node of the form ¼ translates to ,

where

is the variable labeling the node

Ò ½

¼ are the variables labeling the child nodes

is the relation defined by

´ µ  ´ µ

Ò ½ ¼ Ò ½ ¼ Ò ½ ¼ iff for all

It is worth pointing out that translating the system in Figure 5.2 to a CSP in the way just described enhances the opportunities for parallelism in propagation beyond those already present in System (5.3). To investigate these would take us beyond the limits of this thesis. 5.2 Interval constraints and interval constraint solver 66

However, it will be useful here to highlight the structure that gives rise to these opportunities by means of a hardware metaphor. In the first place it is important to note that the sets of constraints arising from the same expression form a constraint system for the purposes of propagation. For example, with the exception of the root, the internal variables only have unique occurrences. As a result, when the DRO of a constraint is activated, it is usually the case that only constraints generated by the same expression are added to the active set. However, the external variables may occur in all of these constraint systems. For the purposes of a parallel algorithm it is useful to imagine the constraint systems arising from each of the expression as hardware “cards”, each connected to a “bus”, where the lines of the bus represent the external variables shared by more than one expression. Whenever two external variables belong to the same equivalence class, their equality is expressed by connecting them by a “jumper” in the hardware model. Such a hardware model is shown in Figure 5.3.

Cards

yn

y2 y1

Bus

Jumper x1 x2

xn

Figure 5.3. Hardware model for a CSP.

The hardware architecture suggests a parallel process for each card that asynchronously 5.2 Interval constraints and interval constraint solver 67

executes DROs of constraints only involving internal variables. The processes synchronize when they access one of the bus variables. The DROs of allEq constraints can also be executed by a parallel process dedicated to each.

Evaluating an expression using interval constraints Consider a term that can be

evaluated in interval arithmetic, and that can be translated into an ICSP as described earlier.

Let us compare the interval arithmetic evaluation of and the result obtained through

GPA applied on . The following lemma appears substantially as Theorem 2.6 in [9].

Lemma 1 Let be a term that can be evaluated in interval arithmetic. Let the variables of

Ò

be ½ . Let be the variable associated with the root of the tree form of . Let be

Ò ½ Ò

the ICSP that results from translating , where the domains of ½ are

·

and where the domains of the internal variables are  . After termination of GPA applied to , the domain for is the interval that results from interval arithmetic evaluation

of .

Lemma 1 is useful in showing that, in the absence of information about the value of the expression, propagation does the equivalent of interval arithmetic. GPA ensures the construction of a fair sequence of DROs in a way that works for all CSPs, irrespective of their provenance. As a result, in GPA some DROs have no effect, and could have been skipped. From Lemma 1 we can deduce that using GPA we never obtain worse results, and, in some cases, we obtain better results. However, unmodified GPA will give the better results in an inefficient way. In the next chapter we will propose several modifications of GPA to obtain the better result in a more efficient way.

Solving a single inequality using interval constraints Let be an ICSP resulting from

translating

´ µ ¼

½ Ñ 5.2 Interval constraints and interval constraint solver 68

Let be the variable labeling the left child of the root; that is, the variable representing the

Ñ ½ Ñ

value of the left-hand side. Let ½ be the domains in of , respectively.

 · The initial domain of is .

Now suppose that GPA is applied to . One possible initial sequence of DRO activa-

¼

tions is the equivalent of interval arithmetic evaluation of the left-hand side, leaving

 

as the only constraint in the active set with the domain for equal to
, the value in

´ µ

½ Ñ

interval arithmetic of
.

¼

At this stage the DRO for is executed. We distinguish the following cases:

¼

. This causes failure to occur.

¼

. In this case, the domain for is unchanged. Therefore, no constraint is added

to the active set. Termination occurs with nonfailure, and is proven to be totally

 Ñ

consistent on the box ½ .

¼

. In this case, the domain for shrinks: the upper bound decreases from

¼

to . This causes the constraints to be brought into the active set that correspond to nodes at the next lower level in the tree. This propagation may continue all the

way down to the lowest level in the tree, resulting in shrinking certain domains of

Ñ ½ .

Let us compare the behavior of GPA with the use of interval arithmetic to solve the same inequality. In the first two cases, GPA gives gives no more information than interval arithmetic: failure or nonfailure. It also does no more work.

In the third case, however, GPA may give more information than interval arithmetic: in

Ñ addition to the nonfailure outcome, it may shrink the domain of one or more of ½ . This is beyond the capabilities of interval arithmetic, which is restricted to transmit infor- mation about arguments of a function to information about the value of the function. It cannot transmit information in the reverse direction. To achieve this extra capability, GPA needs to do more work than the equivalent of interval arithmetic evaluation.

5.2 Interval constraints and interval constraint solver 69

µ

Floating-point number function squashLftC´

apply propagation on the ICSP with ½ ;

if (result of propagation is failure) return ;



if ( is canonical) return ;

  

midpoint of ;

¼

 ´ µ

squashLftC ;

¼ ¼

if ( ) return ;

µ

return squashLftC´ ;

Figure 5.4. Left bound squashing algorithm using interval constraints.

Relational box consistency In Chapter ¾, we showed how box consistency can be achieved using interval arithmetic. In what follows, we will use interval constraints to reach box con-

sistency. To show how this can be done, consider the ICSP obtained from translating

´ µ ¼

½ Ò

½

½ ½ ¾ ¾ Ò Ò

´ µ ´ µ ½

where is a constant that satisfies lb ½ rb . Suppose we apply GPA to ,

 ´ µ ½ and suppose failure results, then the domain of ½ can be improved to rb . Using this simple idea, box consistency can be obtained by means of interval constraints. In fact, in interval constraints, the left bound squashing algorithm becomes squashLftC given in Figure 5.4.

The box consistency obtained by using interval constraints is called relational box consis- tency [60]. The relational and functional box-consistency algorithms can be improved by means of

our dynamic and adaptive box-consistency algorithms discussed in Chapter . Chapter 6

Dual, clustered, deterministic, and selective propagation algorithms

In this chapter we present a new approach, based on relations and recursive functions on power sets, to treat propagation algorithms. This approach reveals new kinds of propagation algorithms, namely dual, clustered, deterministic, and selective propagation algorithms. The main purpose of these algorithms is to improve GPA, and make it as fast as possible. Dual propagation algorithms are a consequence of the duality between variables and constraints. These algorithms use the variables as the main criterion for grouping the DROs. Instead of only associating a composite DRO with each constraint as done in several im- plementations of propagation algorithms, we also associate a DRO to each pair (constraint, variable). In clustered propagation algorithms, domain reduction operators (DROs) are grouped into several classes, and a specific propagation algorithm is applied on each. This allows us to uniformly treat different kinds of constraint satisfaction problems (hard, discrete, soft, fuzzy, etc.), and to handle user-defined constraints efficiently. Since atomic formulas have structured forms inherited from their expressions, it is use- ful to follow these structures when applying DROs. This leads us to structured propagation algorithms, especially tree-structured ones. To increase the efficiency of GPA even further, we introduce selective propagation al- 6.1 Introduction 71

gorithms in which the information gained about constraints and/or atomic formulas is used to delay the activation of some DROs, or to remove them completely from the active set.

6.1 Introduction

According to Apt’s chaotic iterations framework [3], by applying DROs in a fair order the domains converge to a limit or one of the domains becomes empty. The resulting Cartesian product of the domains becomes the greatest common fixpoint of the DROs. If one of the domains becomes empty, it follows that no solutions exist within the initial domains. Although fairness is enough to reach the fixpoint, it is not enough to produce fast algorithms. In fact, a fair sequence of DROs may contain DROs that do not have any effect. In addition, for most ICSPs, how quickly a fixpoint is reached depends on the order in which the DROs are applied. This is why we propose a different framework based on relations and recursive functions on power sets. Instead of only associating a DRO with each constraint as done in several implemen-

tations of propagation algorithms, we associate a fractional DRO to each constraint and

variable in . This fractional DRO is denoted DRO( , ), and is used to remove all or some inconsistent values from the domain of using . Hence, each constraint has a list

of fractional DROs. Each one of them is associated with a variable in the constraint. An

µ ·  example, the constraint sum´ defined by the relation has the following

fractional DROs:

¼

´ µ´ µ   ´ ´ µµ

DRO sum

¼

´ µ´ µ   ´ ´ µµ

DRO sum

¼

´ µ´ µ   ´ ´ · µµ

DRO sum

where the intervals , and are the old domains of , and respectively, and the

¼ ¼ ¼

intervals , and are their new domains.

µ The constraint leq´ defined by the relation has the following fractional 6.2 Relations and Recursive Functions on Power Sets 72

DROs:

¼

´ µ´ µ   ´  ´ µµ

DRO leq rb

¼

´ µ´ µ   ´  ´ µ ·µ

DRO leq lb

µ  Another constraint that we shall make use of is int´ defined by the relation .

This constraint expresses the fact that the variable is an integer, and allows us to handle

µ

discrete CSPs. The constraint int ´ has only one fractional DRO given by

¼

´ µ´ µ   ´  µ

DRO int

¾ Some discrete examples are given in Section of Chapter .

6.2 Relations and Recursive Functions on Power Sets

In what follows, a relation on a sequence of sets is a subset of their Cartesian product.

´ µ

Let and be two sets. Given a relation ½ on and an element in ,we

define the range of by

´µ  ´ µ ½

½ .

Given an element in , we define the domain of by

½

´ µ  ´ µ ½

½ .

´ µ ´ µ ¾

The composition of two relations ½ and on and respectively is a

´ µ Æ ¾

relation on denoted by ½ , and defined by

Æ  ´ µ ´ µ ´ µ

½ ¾ ½ ¾

´ µ

A relation on is said to be an equivalence relation if and only if it is:

´ µ

1. Reflexive, that is, for every in ,wehave is in .

µ ´ µ

2. Symmetric, that is, if ´ is in ,sois .

µ ´ µ ´ µ 3. Transitive, that is, if ´ and are in ,sois .

6.2 Relations and Recursive Functions on Power Sets 73

´ µ

Given an equivalence relation on , the equivalence class associated to an element

of is a subset of defined by

 ´ µ

Note that the equivalence classes form a partition for .

 Ò

Given a CSP with the set of constraints ½ , and with the set of variables



Ñ Ú 

½ , we define the relations and as follows:

´ µ

Ú is a relation on and is given by

 ´ µ




Ú and are involved in the same constraint

´ µ

 is a relation on and it is given by

 ´ µ




 is involved in the constraint

´ µ

Ò¾Æ

Theorem 12 The sequence of relations Ò defined recursively by



¼ Ú

 Æ

Ò·½ Ò Ú

is constant after a finite positive integer . Moreover, the relation is an equivalence

relation. If the number of the equivalence classes associated with is , then the CSP

can be decomposed into independent CSPs.

The sequence of relations defined in this theorem allows us to decompose a CSP into independent CSPs, when this is possible. This sequence also permits us to find the number of steps necessary for a change of a variable to affect another variable as stated in the

following theorem.

´ µ




Theorem 13 Given two variables
and in , the smallest integer for which

 Ò is the number of steps it takes for the changes of to affect .

Let us now introduce our approach to propagation algorithms.

6.3 Dual Propagation Algorithms 74

©

Let  be a function defined recursively on the sets of constraints of as follows:

© ´µ 



Ë

½

´ µ © ´ µ  ´µ




 where is a variable in with a domain that

Ü

´ µ

is reduced by DRO

µ © ´ µ  © ´ µ © ´






Ò

½ ½ ¾

 ©

In other words, the function  takes a set of constraints, and if this set is empty, its value is

© 

empty set. If the set is a singleton containing the constraint
, then the value of is the set

of constraints involving a variable for which the domain is reduced by the DRO(
, )(

© ©  is not included in this set). Otherwise, the function  returns the union of the values of

on a randomly selected constraint, and on the rest of the constraints. Note that in the case

© ´ µ 



where the DRO of a constraint
does not affect the current domains, then

Ò

´© µ

 ¾Æ Theorem 14 The sequence Ò converges to the empty set. This limit is reached for

a finite .

This theorem is an immediate consequence of having a finite number of floating-point numbers. Note that when the sequence reachs its limit, the domains attain their fixpoint.

A possible implementation of propagation algorithm that directly follows from the def- ©

inition of  is shown in Figure 6.1.

It is easy to see that this is a recursive version of the generic propagation algorithm shown in Figure 5.1. This way of looking at the propagation is used in the next section to present dual propagation algorithms.

6.3 Dual Propagation Algorithms

 Ò

In what follows, we consider a CSP involving a set of constraints ½ and

 Ñ a set of variables ½ . A question that arises from the previous section is

Instead of using a function on the power set of the constraints, can we use a function on the power set of the variables?

6.3 Dual Propagation Algorithms 75

1. Set propagate(Set of constraints )

 2. if ( )

3. return ;

4. if ( is a singleton containing
)

½

´ µ

5. for (every variable in  )

6. apply DRO(
, );

½

´ µ ´µ



7. return the union of  , where is a variable in for which

the domain was reduced by DRO(
, );

8. let  be a constraint in ;



9. return the union of propagate(  ) and propagate( );

Figure 6.1. A propagation algorithm based on recursive functions on power sets.

To answer this question, consider the function defined on the power set of the variables as

follows

© ´µ 

Ú

Ë

½

´ µ © ´ µ  ´ µ


Ú


where is in





´ µ

and the domain of
is reduced by DRO

© ´ µ  © ´ µ © ´ µ

Ú

Ú
Ú

Ò

½ ½ ¾

 ©

This function was obtained from  of the previous section by substituting every oc-

currence of a constraint with a variable This explains the use of the term “dual”. Note also

© ´ µ 

Ú
that if the domain of a variable
is not affected by any DRO, then .

As done in the previous section, we have the following theorem.

Ò

µ ´©

Ò¾Æ Theorem 15 The sequence Ú converges to the empty set. Moreover, this limit is

reached for a finite .

A possible implementation for the dual propagation algorithm is given in Figure 6.2.

6.3 Dual Propagation Algorithms 76

Set Dpropagate(Set of variables )

 if ( )

return ;

if ( is a singleton containing
)

´ µ

for (every constraint in  )

´ µ

apply DRO
;

½

´µ ´ µ



return the union of  , where is in

´ µ

and the domain of
has changed by DRO ;

let  be a variable in ;



return the union of Dpropagate(  ) and Dpropagate( );

Figure 6.2. Dual propagation algorithm based on recursive functions on power sets.

A non-recursive version of the dual propagation algorithm is shown in Figure 6.3.

A question that arises is “Why did we look for dual propagation algorithms?” The reason

is that the generic propagation algorithm is not optimal in several ways. First of all, each time a constraint is activated, all the DROs associated with the variables of are applied hoping to reduce the domains of these variables. In general, this is not efficient; only some domains of the variables may change. In the dual only the DRO that contracts the current variable is used. Second, the generic propagation algorithm does not emphasize the idea of a variable being constrained by a set of constraints. The dual does this by intersecting all the DROs of the constraints involving the same variable. Last, such an intersection may save the number of operations needed to fully reduce the domain of a variable. In the generic propagation algorithm this intersection is done separately and in a pairwise manner. 6.4 Clustered Propagation Algorithms 77

1. put all variables into the active set ;



2. while ( )

3. choose a variable from ;

´ µ

4. for (every constraint in  ) apply DRO( , );

5. if (one of the domains becomes empty) stop; 6. add to any variable involved in any constraint containing , if the domain

of has been reduced;

7. if the domain of was not changed, then remove from ;

Figure 6.3. The dual generic propagation algorithm.

6.4 Clustered Propagation Algorithms

The generic propagation algorithm and its dual can be seen as two different ways of clus- tering fractional DROs; the first algorithm uses constraints, and the second uses variables.

Using our framework, we can form others as explained below.

Let us go back to the algorithm shown in Figure 6.1. From the lines  and , it is easy to see that we selected a single constraint and used it in propagate(). As far as recursion is concerned, there is no difference between choosing only one constraint or choosing several ones. The algorithm given in Figure 6.4 shows how recursion can be nicely used on several

sets of constraints.  This algorithm is the same as the algorithm given in Figure 6.1 except for the lines and in which the constraints are grouped into clusters, each viewed as a sub CSP of the original CSP, and propagate() is called on each. This idea of clustering can be used to implement distributed constraint propagation al- gorithms [44]; each cluster is handled by a different CPU. Another important consequence of clustering is to allow different CSPs with different consistency algorithms to be com- bined. In addition, each atomic formula typically contains nested terms, and, thus, trans-

6.4 Clustered Propagation Algorithms 78

1. Set propagate(Set of constraints )

 2. if ( )

3. return

4. if ( is a singleton containing
)

½

´ µ

5. for (every variable in  )

6. apply DRO(
, );

½

´ µ ´µ



7. return the union of  , where is a variable in for which

the domain was reduced by DRO(
, );

×

8. partition into clusters × ;

Ñ

½

×

9. return the union of propagate( × ), ..., and propagate( );

Ñ

½

Figure 6.4. A clustered propagation algorithm based on recursive functions on power sets.

lates into several constraints. By clustering these constraints based on the atomic formula they originated from, the structure of the CSP, as specified by user, is kept intact. In other words, the gap between the modeling and the implementation can be minimized using the clustering technique. In fact, it is usually the case that a problem is decomposed into sev- eral sub-problems. Each sub-problem generates a set of constraints that model it. Using the clustering technique, the user can keep this decomposition in the constraint level. Note that clustering is supported by the fact that CSPs have a hierarchical structure and can be composed. Precisely, a composite CSP can be viewed as a set of clusters, each corresponding to a CSP. We call special attention to the fact that the clustering technique can be also used for variables instead of constraints. That is, the algorithm given in Figure 6.2 can be rewritten in such a way that the variables are partitioned into several clusters on which propagate() is recursively applied. 6.5 Deterministic propagation algorithms 79

6.5 Deterministic propagation algorithms

Propagation algorithms do not specify the order in which the DROs are applied. Thus, they are not deterministic. In order for propagation algorithms to follow the structure of an atomic formula, and mimic interval arithmetic evaluation, they need to be deterministic. By this we mean that an order should be introduced when applying DROs. This can be done in two ways. The first way is by changing the data structure used to hold constraints and variables. So, instead of using set as our data structure for holding constraints or variables, we need to deploy a sequence container such as list, and queue. Thus, the active set in GPA becomes an ordered data structure in which the next fractional DRO to activate is not randomly chosen but imposed by the order of the data structure.

The second way is by introducing a choice function that takes a set of fractional DROs and picks the next fractional DRO to activate from it. The clustered and deterministic propagation can be combined to efficiently handle a system of equations. By treating each equation as a cluster, and applying structured prop- agation on each cluster, we most likely get better time-efficiency than using GPA only. As

an illustration let us take the following system of equations:

¾ ¾

·  ½¼

¾



½¼ ½¼

where and are both in .

To keep the same structure of both equations, we consider two clusters: ½ that only

¾



contains the primitive constraint , and ¾ that contains the primitive constraints

¾ ¾

 ·  ½¼

that originate from the decomposition of the atomic formula , namely

¾ ¾

  ·  ½¼

. The cluster ¾ is structured into a tree, given by the tree

form of the formula , so that we can apply structured propagation to it. The whole ICSP

¾ can be represented as a set with the two clusters ½ and . 6.6 Selective propagation algorithms 80

Note that structured propagation is the one used to solve a single inequality discussed

in Chapter ¾.

6.6 Selective propagation algorithms

In some cases, we know from previous computations or other sources that the DROs of certain constraints can be removed from the active set either temporarily or permanently without affecting the outcome of a propagation algorithm. This information must be used

to speed up the algorithm. As a first case, suppose that an atomic formula , which belongs to an ICSP , is known

to be totally consistent on a box . If the DROs of the primitive constraints originated

from the decomposition of are removed from the active set, then the outcome of any propagation algorithm will not be affected. Thus, our first selective rule is

The DROs that come from translating a totally consistent atomic formula into constraints must be removed from the active set permanently.

As another case, consider the following scenario, which happens frequently during the

search process. Suppose we apply GPA to an ICSP . After the termination of GPA,

suppose the domain of is changed to an interval that is a proper subset of it. If we apply GPA to with an active set initialized with the primitive constraints only involving , then GPA terminates with the same result as when the active set would have been initialized to contain all constraints. This observation follows immediately from the idempotency of the DROs. So our second selective rule is

The previous results of propagation algorithms must be used to selectively ini- tialize the active set.

This rule can be generalized to Selective Initialization discussed in [61].

6.7 Implementation

The architecture used in our implementation is shown in Figure 6.5. The hardware layer deals with the rounding modes and the choice between big-endian and little-endian ar- 6.7 Implementation 81

LAYER 3 Variable Constraint ICSP LAYER 2 Interval arithmetic IA LAYER 1 Floating-point arithmetic FLPT LAYER 0 Hardware

Figure 6.5. Architecture of our implementation.

chitecture. The floating-point arithmetic layer implements the floating-point arithmetic according to IEEE 754 standard. Our implementation supports both single and double pre- cision floating-point numbers. Some constants and helper functions are also coded in this layer. The actual implementation of the hardware and the floating-point arithmetic layers consists of a C++ namespace containing several functions and constants that are relevant to the interval arithmetic layer. This layer encompasses the interval arithmetic operations

listed below.

·

, , ,



, , .

ÔÕ Ü

ÐÒ ´µ Power function: , exponential function: , logarithmic function:

min and max functions

ØÓÁÒØ The function that takes an interval and returns the largest interval with integer

bounds in .

Since a CSP is based on the notions of variable and constraint, we represented them in the variable-constraint layer. The implementation of all the layers shown in Figure 6.5 is mainly based on three C++ classes: Interval, Variable and Constraint. These are described next. The Interval class represents a domain of a variable, and contains the elementary oper- ations on intervals listed above. This class implements the IA layer. 6.7 Implementation 82

Interval -lft : double -rght : double +add(in X : Interval, in Y : Interval) : bool +sub(in X : Interval, in Y : Interval) : bool +mult(in X : Interval, in Y : Interval) : bool +div(in X : Interval, in Y : Interval) : bool +leq(in X : Interval) : bool +geq(in X : Interval) : bool +exp(in X : Interval) : bool +log(in X : Interval) : bool +power(in X : Interval, in p : int, in q : int) : bool +toInt() : bool 1

Variable Constraint -domain : Interval * * * +shrinc() : bool +shrinc() : bool * +shrinc(in V : Variable) : bool *

1 1 1 TreeConstraint Sum Prod Log Leq Eq ICSystem IVSystem

Figure 6.6. UML diagram for our implementation.

The Variable class represents an unknown, and has a method called shrinc() that reduces the domain of the variable based on the constraints in which it is involved. This method is used to implement the generic dual propagation algorithm. The Constraint class is an abstract class that represents a “constraint” in a CSP. It has

two virtual methods shrinc(), which shrinks all the domains of the variables involved in the constraint, and shrinc(Variable ), which shrinks only the domain of the variable (i.e. the

fractional DRO associated with the constraint and ). The Variable and Constraint classes implement the ICSP layer. A UML diagram for our implementation is shown in Figure 6.6. The classes TreeCon- straint, ICSystem, and IVSystem shown in the UML diagram are discussed next. The TreeConstraint class is used to present the tree structure of an atomic formula. The shrinc() method of this class uses structured propagation to go up and down the tree until no domain changes. The ICSystem class is a generic CSP that holds other CSPs and/or atomic formulas, and applies propagation algorithms on them. Note that this class calls the shrinc() method on 6.7 Implementation 83

its constraints, and not shrinc(Variable ). The IVSystem class can be viewed as the dual class of ICSystem. The shrinc() method of this class calls the shrinc() method of its variables. Thus, the shrinc() method of IVSystem is an implementation of the dual generic propagation algorithm. We call special attention to the fact that for implementing TreeConstraint, ICSystem, and IVSystem we used the composite design pattern [15]. This pattern allows us to treat a cluster of constraints as if it were a primitive constraint with shrinc() method as its DRO. Thus, the user can group some constraints in his ICSP into several clusters, and treat each cluster as an independent ICSP that has its own shrinc() method. The shrinc() method of this ICSP is called as if it were a primitive constraint. To simplify the modeling of CSPs, our implementation offers a graphical user interface layer on top of the variable-constraint layer. Figure 6.7 shows how the previous example is coded using this layer. The same figure also shows the CSPs used to model the user’s CSP (left panel) as well as the values of the intervals after applying propagation (bottom panel). 6.7 Implementation 84

Figure 6.7. A graphical user interface for our implementation showing the modeling (clus- ters of constraints) and the intervals resulted from applying propagation on the user’s CSP. Chapter 7

Dynamic and adaptive box-consistency algorithms

The purpose of this chapter is to present two algorithms for achieving box consistency, namely dynamic and adaptive box-consistency algorithms. In some cases, these two algo-

rithms are faster than the functional and relational box-consistency algorithms given in the

¾¾ previous chapters. The reader may refer to Section of Chapter for some examples.

7.1 Box consistency

As we explained in Chapter ¾, box consistency (BC) selects a variable and a domain bound. Then, it uses binary search to determine a boundary interval that can be shown to contain no solutions. This boundary interval can then be removed from the domain, and thus shrinking the domain. This is repeated until a boundary interval with width less than a certain toler-

ance is found that cannot be shown to contain no solutions. When this is the case for both boundaries of all variables, the domains are said to be box consistent with respect to the

tolerance and with respect to the method for showing inconsistency. When this method is interval arithmetic, we obtain functional BC. When it is propagation, we get relational BC [60]. Figure 7.1 illustrates the steps of narrowing a rectangle using BC. A close look at these steps shows that obtaining BC is time consuming, especially when it approaches the bounds 7.2 Dynamic box consistency 86

of the consistent box. The left and the right squashing functions improve the bounds only

by small amounts, especially when the tolerance is close to zero. Even in the case where

the tolerance is not that close to zero, BC still suffers from another deficiency as explained below.

Y

X

Figure 7.1. Illustration of BC steps. Suppose that BC starts with a variable and its domain . BC tries to remove the

boundary intervals of until a left and a right boundary intervals with width less than a

certain tolerance cannot be eliminated. BC then selects another variable, say , and its

domain , and tries to reduce as it did for . Now, suppose that gets reduced. Then

BC has to go back to to try to reduce its domain. Many boundary intervals on which BC spent a lot of time to fail removing them in the first try can be eliminated easily in the

second try, and after the domain of is shrunk. Thus, focusing and trying hard on one variable at a time is the source of the deficiency of BC, and not the value of the tolerance

as thought by Granvilliers, Goualard and Benhamou [18]. This deficiency is solved by dynamic box consistency algorithm presented next.

7.2 Dynamic box consistency

The deficiency of BC is behind the idea of dynamic box consistency (DBC). Instead of improving only one interval at a time as in BC, DBC goes beyond that to reduce a box by considering all its projections in parallel. DBC reduces the domains of the variables in a cyclic manner. Figure 7.2 shows this process in the case of two variables. 7.2 Dynamic box consistency 87

Y

X

Figure 7.2. Illustration of DBC steps.

The power of DBC is based on reducing the box not from one direction (i.e. projection)

at a time but from all directions at once. That is at each step , the box Ò is produced

½

½ Ò ½

from the box Ò at step by improving a projection of . The projections are

´ µ

Ò¾Æ

processed in a cyclic manner. The sequence of the boxes Ò converges to a fixpoint , where is box consistent.

Figure 7.3 and Figure 7.4 shows clearly the difference between the boundary intervals removed by BC and DBC in the case of two variables and .

x y

Figure 7.3. Illustration of the boundary intervals removed by BC.

From these two figures, it is worth noting that DBC slices all the sides of the box at once as opposed to BC that only slices one side of the box at a time. This implies that DBC throws away large boundary intervals before small ones. Thus, the power of DBC over BC. To attain box consistency in a balanced fashion, we propose the algorithm DBC pro-

vided in Figure 7.5.

 ¼ Let us describe what the algorithm does in the case where . 7.3 Adaptive box consistency 88

y

x

Figure 7.4. Illustration of the boundary intervals removed by DBC.

DBC is built upon BoxNarrowing() function that takes a set of intervals representing peripheral intervals that covers the box to be improved; they are the selected left and right

intervals of all projections. These intervals are putting into a queue . We can then choose

a strategy to order the intervals in , for example, put the largest intervals into the front.

If the intervals in the queue are canonical, then box consistency is attained, and the box

 is returned. Otherwise, a splittable interval
 is removed from , and is split. Since

can be the left side or the right side of the th projection, two cases should be treated. If the interval is a left interval, then the left narrowing is used. Otherwise, the right narrowing is applied. It is worth noting that in DBC algorithm the whole box is reduced at each step. This is not the case with BC that reduces only one side of an interval at each step.

7.3 Adaptive box consistency

In the case of BC, and DBC, we first check if an interval can be removed as a whole. If not it is bisected, and its halves are subjected to the same test. This way of reducing the box can be viewed as an “eager reduction” in which we try to remove bigger parts first. As we can easily deduce, when a CSP has a solution near the bounds of the box, such a reduction 7.3 Adaptive box consistency 89

let be a given ICSP;

let be the tolerance used;

 Ñ

let ¼ be the initial box; propagate on ;

if (propagation fails) return ;

¼ ½ ½ Ñ Ñ

else return BoxNarrowing( ¼ );

´ µ

Ð ¼Ö ½Ð ½Ö ÑÐ ÑÖ

Box function BoxNarrowing ¼

Ð ¼Ö ½Ð ½Ö ÑÐ ÑÖ

let be a queue of ¼ ;

if (intervals in are of width less than all intervals are canonical) then return ;

while ( is not empty)

remove from the queue a splittable interval
 ;

Ð
Ö split
 into and ;

if (d == l)

Ð propagate on with
;

if (propagation fails)

´ µ ´ µ
Ð

change the box by updating lb
to rb ;


Ö

Ð ;

else


Ð

Ð ; else

(Continued next.)

7.3 Adaptive box consistency 90

Ö propagate on with
;

if (propagation fails)

´ µ ´ µ
Ö

change the box by updating rb
to lb ;


Ð

Ö ;

else


Ö

Ö ;

¼

´ µ 

Ð ¼Ö ½Ð ½Ö ÑÐ ÑÖ

BoxNarrowing ¼ ;

¼

for (every projection of )

´ ´ µ  ´ µµ
Ð

if lb
rb


Ö
Ð ;

changed = true;

´ ´ µ  ´ µµ
Ö

if rb
lb


Ð
Ö ;

changed = true;

´ µ

Ð ¼Ö ½Ð ½Ö ÑÐ ÑÖ

if (changed ==true) return BoxNarrowing ¼ ; ¼

return ;

Figure 7.5. Structure of dynamic box consistency algorithm with a tolerance . 7.4 Other improvements 91

is not efficient because the actual reduction of the box happens late. That is, the boundary intervals are not removed as early as possible. To speed up the reduction process, an alternative way is to use a “lazy reduction” in which we try to remove a smaller boundary interval first. If we succeed, then we make the next boundary interval bigger, and try to eliminate it. If we did not succeed, we make it smaller, and try again. With this process we can make the reduction more productive by removing the parts of a box as soon as possible. This way of achieving box consistency is called adaptive box consistency (ABC). The qualification adaptive is used in the sense that under this algorithm the width of the interval boundary tried is decreasing and increasing: it varies between being big, when the previous box is removed, and being small when it is not. In other words, the width of the interval boundary changes with the outcome of the previous operation of ABC,as opposed to the other algorithms in which the width of the interval boundary tried is all the time decreasing. An implementation of ABC is shown in Figure 7.6.

In Figure 7.6 we only gave the pseudo-code for ABC that shrinks the left of the domain of

a variable . The pseudo-code for shrinking the right can be easily deduced. It is worth noting that ABC is similar to BC in the sense that it focuses on one variable at a time. Thus, ABC can be improved by using the idea behind DBC. By combining ABC and DBC, we get an adaptive dynamic box-consistency algorithm (ADBC).

7.4 Other improvements

As the reader may have noticed, relational box-consistency algorithms make use only of the failure state of propagation. To take full advantage of propagation, we should also use

the other states.

During squashing, it can happen that applying propagation on with ½ terminates

with success. By this we mean that propagation proved that is totally consistent on the

Ñ ½ box ¾ . In this case, we should try to reduce only the other side of ,

7.4 Other improvements 92

 

Floating-point number function ABCL (variable , int , interval , tolerance )

  ´ µ

set an interval to the left part of of size ;

apply propagation on with in ;

if (failure results)

 ´ µ

rb ;

µ

if ´ return ;

 ¾

set to a smaller value (e.g. );

  µ

return ABCL´ ;

´ µ

if ( is canonical ) return ;

 ¾

set to a bigger value (e.g. );

  µ

return ABCL´ ;

Figure 7.6. A pseudo-code for adaptive box consistency algorithm with a tolerance .

7.4 Other improvements 93

 ¾

since all the elements of
for are supported by an element of , which is a subset of ½ .

In any functional or relational box-consistency algorithm, if a domain of a variable,

¾ Ñ

say ½ , is changed, then all the other variables, namely , are reconsidered

for squashing. So, even when a variable, say Ñ , cannot be affected by the changes of

Ñ

½ , the algorithm tries to reduce ’s domain. To avoid useless squashing, we must only

try reducing the domains of the variables involved in the same constraints as ½ . In the

½

´ µ

½ ½ notations of the previous chapter, the set of variables to reconsider is Ú . This

suggest the use of an active set that stores the variables to consider for squashing. Each

½

´µ

time we succeed to reduce the domain of a variable , we add the elements of Ú to . The process is repeated until is empty. So, the structure of the algorithm is similar

to the dual propagation algorithm. To improve the algorithm further, when the domain of a

´µ variable changes, we apply propagation on with an active set initialized to  , before going to squash the domain of the next variable. As shown in the last chapter, this can help reduce enormously the domains of several variables at once instead of squashing one at a time.

With the exception of DBC, all functional and relational box-consistency algorithms

Ñ use a round-robin strategy on the variables: given the variables ½ , the algorithms tries to reduce their domains in a specific order, and repeat the process until the squashing

is done. By introducing the active set , we can choose any order at any time. To illustrate how easy it is to implement the above improvements, we refer the reader to the algorithm in Figure 7.7.

7.4 Other improvements 94

Ñ

initialize the active set to ½ ;

while ( is not empty) remove a variable from ;

if (during squashing, propagation succeeds on a part of )

return;

if (during squashing, the domain of is reduced)

´µ

apply propagation on with an active set initialized to  ;

if (the domain of is reduced)

½

´µ

add to all the elements in Ú ;

else

½

´µ

add to all the elements in Ú ;

Figure 7.7. An improved box-consistency algorithm. Chapter 8

Global optimization via interval constraints

As pointed out in Chapter , interval constraints can simulate interval arithmetic. So, we could redo all interval arithmetic algorithms for global optimization using interval con- straints, but we can do better. In fact, its ability to shrink domains using propagation makes it a more powerful tool than interval arithmetic [9, 27]. After reviewing the general structure of the interval constraint algorithm for global optimization, we show how to improve the algorithm using the “exploration” idea. We then discuss the necessary changes to make the algorithm an efficient tool for verifying point methods for global optimization. We conclude the chapter with some remarks on using “exploration” to improve interval constraint solvers and squashing algorithms.

In this chapter, we consider the global optimization problem (1.1) where the feasible

region is expressed as an ICSP  , and the objective function can be decomposed into constraints for which domain reduction operators are available.

8.1 Overview

The problem (1.1) can be written as

Ò

Ê Ê ´µ 

and  (8.1)

´µ  

where is the vector of variables in  . Let be the ICSP formed by and ,

Ò

  ·  Ê where is a variable with a domain , and the domain of is .

8.1 Overview 96

8.1.1 Computing a lower bound for Ñ
Ò

It is obvious that the left bound of is a lower bound for Ñ
Ò . So, is a lower bound

for Ñ
Ò . But if we could shrink , we would obtain a better lower bound. Therefore, we

could apply propagation on . However, by using one of the left-squashing algorithms on

(ABCL given in Figure 7.6 or squashLftC given in Figure 5.4), we obtain a much better lower bound. If the global optimization problem (8.1) is constrained, then we may obtain

an even much better lower bound by running one of the squashing algorithms on all the

Ñ
Ò

domains. Moreover, in case where we know an upper bound Ù for , we can further

´µ Ù

improve the lower bound by adding Ù (or, equivalently, ) to the ICSP .

8.1.2 Computing an upper bound for Ñ
Ò

Ñ
Ò

In interval arithmetic, an upper bound Ù for is used only to eliminate any box for

´ µ

which the lower bound of is greater than Ù . So, the whole box is either eliminated or

kept to be bisected. Although we may have a good upper bound for Ñ
Ò , it may not help

because the elimination depends on the sharpness of the lower bound of at the box .

´µ

In interval constraints, by adding Ù to the ICSP , and applying squashing on

all the domains (or propagation on ) we can shrink a box that we could not eliminate. The

better the upper bound Ù is, the better the box shrinks, and the better the lower bound of

Ñ
Ò gets. Thus, having a good upper bound is of great importance in interval constraints.

In fact, the efficiency of the interval constraint algorithm for global optimization depends

strongly on the upper bound for Ñ
Ò .

To find an upper bound for Ñ
Ò , we proceed as follows.

If a box is proven to contain a solution to  , then applying squashing on all the

domains of the variables in with gives an upper bound for Ñ
Ò (which is the right

bound of the domain of ). In particular, when a point in is a solution to  , using

the box instead of gives us a better upper bound. The point is usually taken as

the midpoint of . We refer to Section 8.4.3, for a discussion on how to prove that a box contains a solution of an ICSP. 8.1 Overview 97

If no point is a solution, we use the upper bound obtained from a box proven to contain a solution. If such a box could not be found, several authors set the upper bound to ·

(see for example [9, 27]). However, we do not agree with such a choice. In fact, the correct

Ñ
Ò

form of the phrase “ Ù is an upper bound for the global minimum ” is:

Ù Ñ
Ò

if the global minimum Ñ
Ò exists, then is an upper bound for it (i.e.

Ù ).

Ò

Ê

Now, suppose that after running a squashing algorithm on with and

 ·

, we get a box for , and an interval for . It is obvious that: if the global

´ µ ´ µ

minimum Ñ
Ò exists, then rb is an upper bound for it. So, we can safely say that rb

is an upper bound for Ñ
Ò . Because of the monotonicity property of interval constraints,

this upper bound can be improved by splitting as we shall see next.

8.1.3 Branch-and-bound in interval constraints After propagation and/or squashing, it usually the case that the domain of is still wide.

To extract more information about the global minimizer, it is necessary to split . The split

is usually done in the middle of the largest component of .

Ð

As a result of splitting into Ö and , we get two optimization problems

Ê ´µ   Ö and (8.2)

and

Ê ´µ   Ð and (8.3)

As we did for the optimization problem (8.1), we compute lower and upper bounds for the generated problems. If the lower bound of a problem is greater than the upper bound found

so far, we discard it. Otherwise, we store it in a list .

Ñ
Ò Ø

The upper bound Ù for is updated as follows. If the upper bound of a problem

 ÑÒ

Ù Ù Ø

was due to a box proven to contain a solution to  , then . If no box can be proven to contain a solution, then Ù is the greatest of the upper bounds of the problems

in . 8.2 Exploration and interval constraints for global optimization 98

By repeated splitting, the lower and upper bounds for the problems in get sharper, and the union of the domains of in the problems in gives better information about the

global minimizer. Note that at each step of the algorithm, the least of the lower bounds of

the problems in is a lower bound for Ñ
Ò .

It is worth mentioning that the efficiency of branch-and-bound does not depend on the sharpness of the lower bound of Ñ
Ò only but also of its upper bound.

8.2 Exploration and interval constraints for global optimization

8.2.1 The fundamental rule of P-NP

As the reader may have noticed, we tried to delay bisections as much as possible. The reason is that propagation and squashing take polynomial time. As soon as bisection is introduced, the interval constraint algorithm for global optimization tends to slow down exponentially with respect to the depth of the search tree. This idea of delaying any com- putation that can exponentially slow down an algorithm is so important that we make it a rule, which we will refer to as the fundamental rule of P-NP. The rule is stated as follows.

Given algorithms that address the same class of problems, any polynomial- time algorithm must be used before any exponential-time algorithm. When an exponential-time algorithm is finally used, we must still look for opportunities to decrease its running time.

8.2.2 Motivation of exploration

As pointed out in the previous sections, the success of interval constraints for global opti-

mization depends on making the upper bound for Ñ
Ò sharp enough to sufficiently shrink the search space. Therefore, we must make some effort to find the right point or box for the upper bound, and not to randomly pick one as is commonly done. In fact, the probability of choosing randomly a point that is close to the global minimizer is almost zero. Although

we do not know the global minimizer, we need somehow to increase the probability of picking a point or a box that will give a good upper bound for Ñ
Ò . Because of the funda- 8.2 Exploration and interval constraints for global optimization 99

mental rule of P-NP, we have to keep in mind that any computation involved must run in polynomial time.

Since the point methods for global optimization are very efficient and often give a very

accurate approximation of Ñ
Ò and where it occurs, it is recommended to use them to find the right point or box. By doing so, we increase enormously the efficiency of interval constraint algorithm for global optimization. Let us do some rough calculations to see why this the case. The point methods give most of the time very accurate results in polynomial time.

Suppose that this happens roughly ¼± of the time. If we incorporate their results into interval constraints, we increase the speed of the interval constraint algorithm for global

optimization roughly by ¼±. It is clear, from the discussion above, that the main use of the point methods is to efficiently extract some information about the global minimum and where it occurs. This information is then used to guide the interval constraint algorithm for global optimization. In other words, the point methods serve as a way to explore the optimization problem. The result of the exploration is then used by the interval constraint algorithm. This is the reason for introducing terms such as exploration and explorer. To generalize our treatment, we will use explorer to mean any polynomial-time algo- rithm that allows us to extract some information about the problem at hand.

8.2.3 Interval constraints with exploration

Interval constraints with exploration proceeds in the following steps:

1. Exploration step: an explorer is used to extract some information about the global

minimum and where it occurs. This information is then used to find a better upper

and/or lower bound for Ñ
Ò .

2. Reduction step: the bounds found are then used by interval constraints, more pre- cisely in propagation and squashing, to reduce the search space for the global min- imizer. If the search space is still wide, we may either do some extra explorations 8.2 Exploration and interval constraints for global optimization 100

(by going back to the exploration step) to find even better bounds or split the search space.

3. Branch-and-bound step: if after the exploration and reduction steps the search space is still wide, we must split it to generate two optimization problems with smaller search spaces. Because of the fundamental rule of P-NP, the splitting must be done so that the probability of discarding one of the problems generated is high. The three steps are repeated until the size of the search space is sufficiently small.

A diagram of these steps is shown in Figure 8.1.

Search space Reduced search space

Reduction step

Exploration step

Branch-and-bound step

Figure 8.1. A visual for the steps involved in interval constraints with exploration.

Before embarking on discussing how exploration is used in interval constraints for global optimization, it is worth noting that interval arithmetic cannot easily benefit from exploration. Although the bounds found during the exploration step can be used to update the bounds in interval arithmetic methods, reducing the space in interval arithmetic with- out bisection is hard. This follows from the fact that interval arithmetic is unidirectional, 8.2 Exploration and interval constraints for global optimization 101

that is, it only evaluates expressions given the intervals of the variables, and cannot used to

reduce the domains of the variables given an upper bound on the expression.

8.2.4 Effect of exploration on the upper bound of Ñ
Ò

By running explorers, we get points, ½ , that approximate the global

minimizer. Using propagation, we check whether some of these points are solutions to

 . Using propagation, we evaluate the intervals of at these points, and choose the least

upper bound. If this upper bound is less than the upper bound Ù found so far, we use it as the new upper bound for Ñ
Ò . If no point is a solution, we use the upper bound obtained

from a box proven to contain a solution.

´µ

By adding Ù to the ICSP and running propagation or a squashing algorithm,

we end up with a reduced domain for and an improved lower bound for Ñ
Ò .

8.2.5 Effect of exploration on the lower bound of Ñ
Ò

As we explained previously, to find a lower bound for Ñ
Ò , an interval constraint algorithm

for global optimization uses the left-squashing algorithm squashLftC given in Figure 5.4

´µ   on the domain of and the ICSP formed by , Ù , and . This algorithm

uses bisection and propagation to improve the left bound of the domain of . If no information about the global minimum is available, then bisection is the right

choice. But with exploration at hand, this may not be the best strategy. In fact, the value

Ü returned at the end of the exploration step is usually very close to the global minimum

Ñ
Ò Ü

Ñ
Ò . So, we know that the lower bound for must often be close to . That’s why, we

should split not in half but near the value Ü . Therefore, the algorithm to use for computing a lower bound for Ñ
Ò should be similar to the adaptive left-squashing algorithm ABCL given in Figure 7.6.

To explain how the algorithm works, let us suppose that exploration uses as the preci-

sion on the values of , and that the points returned at the end of exploration are solutions

to  .

Let be the point returned at the end of the exploration step. Let the interval of at

8.2 Exploration and interval constraints for global optimization 102

´ µ

Floating-point number function ExplLB Ô

´µ

apply propagation on with Ô ;

if (result of propagation is failure)

¾  ¼

if ( ) return Ô ;

´ ¾µ

return ExplLB Ô ;

¾ Ñ
Ò

if ( Ô is less than or equal to the lower bound of )

return the lower bound of Ñ
Ò ;

´ µ

return ExplLB Ô ;

Figure 8.2. Computing a lower bound for Ñ
Ò using exploration.

´µ   

Ô Ñ
Ò Ô

be Ô . If the explorer is accurate, then must be close to . So, to find

Ô

a lower bound for Ñ
Ò , first, we run propagation on the ICSP formed by and .

 ¾  ¾

If one of the domains becomes empty, then we use . Otherwise, we take .

¾  ¼¼ The process is repeated until . The pseudo-code for the algorithm is given in

Figure 8.2.

¾ Some experimental results related to ExplLB are given in Section of Chapter .

For the sake of simplicity, we took to be the precision during exploration. To be

more effective, must also take into account our confidence about Ô being near the global minimum. When our confidence is high, must be small, and when it is low, must be big. To increase our confidence about the result of an exploration, one could launch several

explorers. In fact, suppose that a point is returned during the first exploration. If all explorers afterwards land at the same point or near it, then is most likely in the global minimizer. The quantification of the confidence concept is not known to us.

8.2.6 Effect of exploration on splitting the search space

Because of the nature of interval constraints, a good upper bound reduces potentially the

¾ ¾ search space (see sections and of Chapter for some experimental results).

8.2 Exploration and interval constraints for global optimization 103

´µ

In fact, by running propagation (or any squashing algorithm) with Ù and ,we

reduce the space containing the global minimizer. If we are confident that Ù is the best

´µ upper bound we can get, then solving the ICSP formed by Ù and (using interval constraints), gives the best approximation for the global minimizer. Otherwise, we can either launch other explorers or split the search space. Running other explorations may not

only help to improve the upper bound, but also improve the tightness of the upper bound

Ù found so far. However, even after several explorations, we may still have wide search space. In this case, we must split the search space. A question that often arises in interval methods is: what is the best strategy to split a search space? The usual answer is to split the largest interval in the middle. The reason behind this strategy is to make the spaces generated as equal and as small as possible so that

the interval value of the function at each sub-space is close to the range of the function

on (this follows from the monotonicity property of interval arithmetic). This, however, must not be the only reason for splitting. According to the fundamental rule of P-NP, when we split a search space, we want also to keep the number of the optimization problems polynomial, that is, we have to increase the probability of discarding at least one of the optimization problems as soon as it is generated. Otherwise, the exponential growth of these optimization problems will limit the speed of the algorithm. This leads us to the main observation about our strategy:

if the splitting is done near the global optimizer, then both optimization prob- lems generated will persist during search. So, to increase the probability of discarding one of the generated problems, the splitting has to be done far from the global optimizer.

Although we do not know the minimizer, we do know an approximation of it through

explorations. So, when splitting, we must make the spaces generated as equal as possible,

as small as possible, and, mainly, one and only one of them has to contain the box Ô with

¾

center and width . This means that we split the largest interval near the midpoint but

outside the box Ô . In this way, we make it more likely that we remove the optimization

8.2 Exploration and interval constraints for global optimization 104

problem for which the search space does not contain the box Ô .

¾ ¾ The reader may refer to sections and of Chapter for some experimental results about our splitting strategy. The architecture of our algorithm is shown in Figure 8.3.

User

Information system

Search space

Exploration

p fex confidence

Interval Constraint Engine Global optimization problem Interval for the global optimum, and boxes for the optimizer

Figure 8.3. Architecture of interval constraints with exploration for global optimization.

A pseudo-code for the algorithm is given in Figure 8.4.

¾ ¾

Some experimental results related to the algorithm 8.4 are given in sections , ,

¾ and of Chapter .

8.2.7 Propagating or squashing?

In each reduction step, we can use either propagation or squashing. Since squashing in- volves propagation, it is obvious that

squashing results are at least as good as propagation, and

squashing running-time is greater than that of propagation.

8.2 Exploration and interval constraints for global optimization 105

Ò

´ Ê ´µ  µ

=reduce  ;

 ´ µ Ñ
Ò

let Ù rb ;//the upper bound for

 ´µ  ´ µ

let  lb ;//the list of optimization problems

 let Ç ;//the list of pairs of the form (box, lower bound)

let Æ be the desired tolerance on the boxes;

while ( is not empty)

´µ 

 Ð remove a pair Ø from ;

//exploration step

´ ´µ  µ

Ø 

Ü =explore ;

if ( is a solution for  )

 ÑÒ ´ ´µµ Ù

Ù rb ;



else if ( Ø is proven to contain a solution for )

 ÑÒ ´ ´ µµ

Ù Ø Ù rb ;

//reduction step

if ( Ù changed)

´ ´µ  µ

Ø Ø Ù 

Ø =reduce ;

´ µ Æ

if ( Ø )

´ µ Ç Ø add Ø lb to ;

else

(Continued next.) 8.2 Exploration and interval constraints for global optimization 106

//branch-and-bound step

ÖØ ÐØ

split Ø into and according to our strategy;

´ ´µ  µ

ÖØ ÖØ Ù 

ÖØ =reduce ;

if ( ÖØ is not empty)

´µ  ´ µ

 ÖØ

add ÖØ lb to ;

´ ´µ  µ

ÐØ ÐØ Ù 

ÐØ =reduce ;

if ( ÐØ is not empty)

´µ  ´ µ

 ÐØ

add ÐØ lb to ;

if (existence of solutions could not be proven)

set Ù to the greatest upper bound of the problems in ;

Ç

let Ð be the least number of the second components the elements of ;

Ç

let Ó be the union of the first components of the elements of ;

 

Ð Ù

output “ Ñ
Ò is in ”; output “the global minimizer is in Ó ”;

Figure 8.4. Interval constraint algorithm with exploration. The function reduce applies

propagation or squashing and makes use of the function explLB. It returns the reduced

domain of , and the interval of . The function explore explores the optimization problem,

´ µ

and returns a point , and a real number Ü . The function returns the width of the

largest component of . 8.3 Interval constraints as a verification tool 107

So, there is a compromise between the efficiency and the sharpness of the bounds. If squashing is used, we risk losing efficiency but we gain sharpness. But if propagation is used, then we risk losing sharpness but we gain efficiency. A solution to this dilemma is based on the following observation.

As the domains of the variables become smaller and smaller, propagation and squashing tend to have the same results.

This implies the following heuristic:

Use squashing when the domains are large. As soon as the domains become small use propagation.

The quantification of “large” and “small” in this heuristic is not known to us. The criterion

that we used in our experimental results is to stop squashing when the widths of the bound-

½¼¼

ary intervals removed are very ½ of the widths of the initial intervals. An example

¾½ about our heuristic is given in Section of Chapter .

8.3 Interval constraints as a verification tool

To show the importance of the exploration idea, we make the following observation:

Interval constraints with exploration for global optimization can be used to verify whether a global optimization algorithm is correct.

To illustrate how this can be done, let us suppose that the algorithm to be verified is called

, and that it returns an approximation Ü of the global minimum, and at least one point

at which Ü occurs. Also, let us assume that the precision used is . In what follows,

Ñ
Ò by “correct” we mean that Ü is within of the global minimum , and that is in the

-minimizer.

The main idea for proving that correct is to use as an explorer. So, we first run

to get Ü and . Then, using propagation, we evaluate the interval of at , which

´µ  ·  ´µ Ü

we denote as . If the interval Ü does not intersect with , then is not correct. Otherwise, we run propagation (or any squashing algorithm) on

8.4 Interval constraint solver with exploration 108

·

with Ü . If the propagation terminates with an empty domain, then is not

correct. Otherwise, using the domains that resulted from propagation, we run propagation

on (or any squashing algorithm) with Ü . If the domain of one of the variables

becomes empty, then is correct. Otherwise, we run the interval constraint solver On

with Ü . If no solution is found, then is correct. If we found a solution such

 that the right bound of the interval of at is less than Ü , then is not correct. In the other cases, we cannot decide. The use of interval constraint solver can enormously slow down our verification pro- cess. So, based on the fundamental rule of P-NP, it is wise to use exploration to extract more

information about the global minimum and the minimizer before using interval constraint

solver. After running propagation (or any squashing algorithm) on with Ü and

, we assume that we end up with a search space . By launching several explorations on

, we can increase our chances to decide that is not correct. Because can be trapped

in a local minimum that is not the global minimum, we may succeed to find a better upper

bound for Ñ
Ò through several explorations. Suppose that during one exploration a point



is found such that the right bound of the interval of at is less than Ü .it

is obvious then that is not correct. If after the exploration we cannot decide whether is not correct, we bisect the search space , and run propagation and exploration on each ICSP generated. So, our interval constraint solver does not only involve bisection and propagation but also exploration. That is why we call it interval constraint solver with

exploration, which is the subject of the next section.

¾½¾ We refer the reader to Section ½¼ of Chapter for an illustration about verify- ing a global optimization tool, namely BARON 7.2, using our software system (BelSyste

described in Chapter ).

8.4 Interval constraint solver with exploration

Let us now show how exploration can be used to improve the efficiency of an interval constraint solver. 8.4 Interval constraint solver with exploration 109

8.4.1 Effect of exploration on splitting and propagation

Interval constraint solvers split the search space at the midpoint of its largest component. This generates two CSPs with equal size and small search spaces. Because of monotonicity theorem of interval arithmetic, it is likely that applying propagation on these two CSPs reduces their search spaces further. However, because of the fundamental rule of P-NP, the goal of splitting must not only be to reduce the sub-spaces obtained but also to avoid the exponential-running time of the solver. Thus, a good strategy for splitting can be easily deduced from the following observation.

If splitting is done at or near a solution, then the two sub-spaces generated will persist during search. So, we need to split far from the solutions.

Since exploration gives some information about the solutions, we can use it to avoid split- ting near a solution. Running explorations first gives us approximations for the solutions. These approximations are then used to guide our splitting strategy: split far from the ap- proximations, so that the sizes of the two sub-spaces generated are almost equal.

8.4.2 Effect of exploration on squashing algorithms

To illustrate how squashing algorithms, namely BC, DBC, and ABC, can take advantage

Ò

of exploration, let us suppose that we have an ICSP with variables ½ with

Ò interval domains ½ , respectively. As we already know, none of the squashing algorithms specifies which variable to start with. In fact, this was one of the reasons for introducing DBC. Knowing which variable will produce the biggest reduction may speed up squashing algorithms. This is where

exploration comes in. It can help us extract the necessary information for ordering the

½
½
·½ Ò ½

variables. In fact, let Ð
be a point in the face with

´ µ ´ µ

the th component lb
.Iflb is , we take the smallest negative floating-point

´ µ

Ð

number instead of lb
. By starting our exploration from , we end up in a point

½
½
·½

with an th component Ð
. By doing the same for the face

´ µ ´ µ ·

½ Ò with the th component rb (If rb is , we take the greatest positive

8.4 Interval constraint solver with exploration 110

´ µ Ö

floating-point number instead of rb
), we obtain an th component . By ordering

´ µ

Ü
Ü

¾½ÒܾÐÖ  the sequence
in a decreasing order, we get the order in which squashing algorithms should proceed. Not only that, it also gives us an idea of how much

we should try to remove first. This can help improve the ABC algorithm. Suppose that the

precision used during exploration is . The amount to try to remove from the left of
is

   · ´ µ

Ð

Ð

lb
. For the right of , the amount to try to remove is rb . In our implementation, we take the midpoints of the faces as the initialization points for

the explorations.

8.4.3 Does a box contain a solution to  ?

In constrained global optimization, being able to prove that a box contains a solution to an ICSP is vital for the sharpness of the upper bound for the global minimum. So, any effort spent on searching for a good upper bound is rewarding. However, the fundamental rule of P-NP forces us to do the search in polynomial time only. For example, we cannot use an

interval constraint solver.

When propagation is applied to the ICSP  with a box , it can be in one of the

following states:

Propagation fails. This implies that  does not have any solution in .

Propagation succeeds. By that we mean that propagation proves that  is totally

consistent on .

Propagation semi-decides, that is, it neither failed nor succeeded.

If propagation succeeds, then the box is proven to contain a solution for  . Unfortu- nately, this case can only happen when  contains only inequalities, and propagation is

able to prove that each one of them is totally consistent on . The cases where equalities are involved or propagation could not prove the total consistency of a box need special

treatment.

Let us first deal with the case where  contains only inequalities, and propagation

could not prove the total consistency on of at least one of them.

8.4 Interval constraint solver with exploration 111

8.4.3.1  involves inequalities only



Suppose that  is box-consistent on . Using an explorer that solves , suppose that

the point returned is (picking the midpoint of is an example of such an explorer). If

is a solution to  , then we are done. Otherwise, we wait until the branch-and-bound step

splits and retry again on each generated box.

¼ ¼ ¼ ¼ ¼

 ´ µ      Ò

Once we find a solution ½ in a box, say ,

½ ½ Ò Ò

 it is of interest to have a larger box Ô containing on which is totally consistent. For

this, we propose two algorithms. The first is called zoomin algorithm, and it proceeds from

¼

a box to its sub-boxes. So, it starts from and moves towards . When a box on which

 is totally consistent is found, we can either return it, or try to enlarge it (using the second algorithm).

The second algorithm is called zoomout algorithm, and treats sub-boxes first; it moves

from to bigger boxes. The zoomin and zoomout algorithms make use of an algorithm, called the stretching

algorithm, similar to the squashing algorithm given in Chapter . The goal of the stretching algorithm is to expand the set of solutions by enlarging the domains of the variables as much

as possible. This leads us to the notion of box satisfaction discussed next.



Definition 2 Let be a tolerance on the domains of the variables of  , and let

¼

   

½ Ò Ò 

½ be a box in . The ICSP is box-satisfied on with respect to

and iff the following two conditions holds:

¼

 



For every in
such that , the state of applying propagation on

    

  

with
and , , is not success.

¼

 



For every in
such that , the state of applying propagation on

    

  

with
and , , is not success.

The ICSP  is box-satisfied on with respect to iff it is box-satisfied on with respect

 ½ to every and . 8.4 Interval constraint solver with exploration 112

let the desired tolerance on the domains;

Float-point number function stretchLft(
)

 
apply propagation on  with ; if (propagation succeeds)

return ;

if ( )

return ;

  

midpoint of ;

¼



stretch(
);

¼



if ( )

return stretch(
); ¼

return ;

Figure 8.5. Left bound stretching algorithm using interval constraints.

An algorithm that stretches a bound to the left is given in Figure 8.5. A similar algorithm

for the right can easily be deduced. These algorithms allow us to achieve box satisfaction

with respect to a variable
and a tolerance . To obtain box satisfaction with respect to all variables, either we use the zoomin or the zoomout algorithm. These are given in Figure 8.6 and Figure 8.7, respectively.

It is worth mentioning that the box obtained after stretching depends on the order in which the variables are stretched. Note that the functional versions of the box satisfaction notion and algorithms can easily be deduced. Note also that dynamic and adaptive versions of the algorithms above can easily be implemented. The notion of box satisfaction leads immediately to box unsatisfaction, which deals with finding a large box that does not contain any solution to the ICSP considered. By

8.4 Interval constraint solver with exploration 113

Box function zoomin(Box )

apply propagation on  with ;

if (propagation succeeds)

return zoomout( );

if (at least one domain of is reduced)

¼

set to ;

´ µ

if ( )

return ;

   

let the largest interval of
be ;

   

Ð Ð

let the largest interval of
be ;

   

Ð Ð

if ( is larger than )

  

midpoint of ;

else

  

Ð Ð Ð midpoint of ;

return zoomin( );

´ µ Figure 8.6. Zoomin algorithm using interval constraints. The function gives the

width of the largest component of .

8.4 Interval constraint solver with exploration 114

    

½ Ò Ò

Box function zoomout(Box ½ )

 ½

for ( to )

¼



stretchRght( );

¼



stretchLft( );

return ;

Figure 8.7. Zoomout algorithm using interval constraints.

changing “succeeds” to “fails” in box satisfaction algorithms, we obtain the algorithms for box unsatisfaction.

Some experimental results related to box satisfaction and unsatisfaction are given in

¾¿ ¾

sections and of Chapter .

Let us now address the cases where  involves equalities as well as inequalities.

8.4.3.2  involves equalities also

Since we do not use any derivatives, we cannot use the method in page ½½ of [9] (it uses Jacobian matrices). If at least two equalities that share at least one variable are present, we

do not search for any solution. But if the equalities involved do not share any variable, and

´ µ  ¼ Ò

have the form ½ where is a continuous function, then using the following

theorem we may succeed to prove that contains a solution for  .

´µ  ¼

Theorem 16 Let be an equality in  , where is a continuous function from

Ê ´µ ¼

¾ ½

to . Let ½ and be two boxes in . If propagation on with

´µ ¼

failed (succeeded) and propagation on the ICSP obtained from with ¾ suc-

¾

ceeded (failed), then , or any box containing non-empty parts from ½ and , contains

´µ  ¼ a solution to .

This theorem follows immediately from the intermediate value theorem.

8.4 Interval constraint solver with exploration 115

¼

find a box on which is totally consistent;

    

½ Ò Ò

let ½ ;

let be the variable with the largest domain;

if (slice( ))

¼

output “  has a solution in ; else

output “Cannot decide”;

Figure 8.8. An algorithm for proving the existence of solutions of  .

¼

Let be the ICSP obtained from the inequalities of  . To prove that contains a

solution to  we proceed as follows.

First, using the previous section, we find a box on which is totally consistent.

´µ  ¼

Then, using bisection, we subdivide into small sub-boxes. Given an inequality

´µ ¼

in  and a sub-box , we apply propagation on with . If propagation

´µ ¼

succeeded or the midpoint of satisfies the inequality , then we record an

(Success). If propagation fails or reduces at least one domain of , we record an

(Failure). Initially, we record a (Undetermined). As soon as we have recorded both

´µ  ¼

and , the equality is proved to have a solution in . The process is repeated for

every equality in  until all equalities are proven to have solutions in , or the sub-boxes are too small. The complete algorithm is shown in Figure 8.8. It uses the algorithm, which

we call the -slicing algorithm, given in Figure 8.9. 8.4 Interval constraint solver with exploration 116

let be an active set;

´µ  ¼

for (every every equality in  )

´µ

insert to ;

boolean function slice(
)

´µ

remove a tuple from ;



if ( )

´µ ¼  

apply propagation on with
;

if (propagation failed or reduced a domain)

 ;

else if (propagation succeeded)



;



if ( )

     

½
½
½

if (midpoint of ½

    ´µ ¼

·½
·½ Ò Ò

satisfies )



;

 

if ( )

´µ put back to ;

if ( is empty)

return true;

if ( )

return false;

  

midpoint of ;

if(slice(
) is false)

return slice(
);

return true;

Figure 8.9. -slicing algorithm. Chapter 9

Experimental results

In this chapter we present some experimental results from my implementation, which I call BelSyste. It grew from code authored by M.H. van Emden. In the first section, we discuss the structure and the main components of BelSyste. We then give some examples to illustrate how our system works. We do not intended to compare our system with other tools, but we shall make some remarks when appropriate. We refer the reader to [41] for a report on the existing tools and their deficiencies.

9.1 Description of BelSyste

9.1.1 BelSyste architecture

To implement interval methods that solve global optimization and constraint satisfaction problems, we used the layer architecture given in Figure 9.1. The architecture is composed of the following layers:

1. Hardware layer - IEEE754: changing the rounding modes to effect the outward rounding is a necessary condition for the correctness of any interval implementa- tion. There are two methods for implementing rounding capabilities: hardware and software. By using the right masks, a floating-point unit (FPU) can be controlled to change its mode. This is the hardware method. The software method involves com- puting the successor and the predecessor of a floating-point number. Although Bel- Syste can use either, we do not recommend the software method because it is slow, 9.1 Description of BelSyste 118

GUI TUI

Parser and lexer

Optimization engine

Interval constraint engine

Interval arithmetic

Hardware layer – IEEE754

Figure 9.1. Architecture of BelSyste.

and the bounds it returns are not as sharp as with the hardware method. In fact, even in the case where an operation on two floating-point numbers gives a floating-point

number , the software method force us to return an interval that has the successor

and predecessor of as bounds. Note also that in the hardware layer big-endian and little-endian are to be treated differently.

2. Interval arithmetic layer: in this layer we define the arithmetic operations on intervals such as addition, subtraction, multiplication, etc.

3. Interval constraint engine: this layer contains the implementation of variables, con- straints, propagation algorithms, and our interval constraint solver.

4. Optimization engine: this layer implements interval methods for global optimization.

5. Parser and lexer: to allow users to enter their equations easily, we added a parser/lexer on top of the previous layers. We used BISON++ and FLEX++ to implement this layer. By specifying the grammar of the inputs, the necessary tokens, and the actions to take when a rule is fired, BISON++ and FLEX++ allow us to produce the necessary files for the lexical analyzer and the parser. The grammar of the inputs in BelSyste is 9.1 Description of BelSyste 119

similar to Numerica’s grammar [27] and to AMPL [14].

6. GUI/TUI layer: Two user-friendly interfaces are implemented in this layer; graphical and textual user interfaces. Among others, the graphical user interface can plot, correctly, a system of constraints with respect to two chosen variables. The reader can refer to [40, 58, 29] for how the plotting can be performed correctly. Qt 3.1 was used to implement the graphical user interface. A textual user interface is also available. By writing a text file that respects the grammar of the inputs, BelSyste can be run from the command line with the text file as an argument.

9.1.2 BelSyste components

The main components of the system are shown in Figure 9.2. The description of each component as is follows.

GUI: contains all the necessary classes for the graphical user interface.

TUI: represents the textual user interface.

PARSER: contains the parser and the lexer.

OPT: the classes for interval methods for global optimization reside in this compo- nent.

BCST: contains the classes BCConstraint, ABCConstraint, and DBCConstraint, which deal with box-consistency.

SBX: has two classes BSat and BUnSat. Starting from a feasible point, BSat tries to find a large box on which the ICSP considered is box-satisfied. BUnSat starts from an unfeasible point, and tries to find a large box on which the ICSP is box-unsatisfied.

CSS: among others, this component contains the following classes: Constraint, IC- System, IVSystem, Function, Interval, TreeConstraint, Solver, VSolver, and Variable.

CPPUNIT: is a C++ unit testing framework similar to JUnit. It is used to test our implementation. 9.1 Description of BelSyste 120

GUI TUI

PARSER

OPT

BCST SBX CPPUNIT

CSS

IEEE754

Figure 9.2. Components of BelSyste and their dependencies. 9.2 Experimental results 121

IEEE754: contains the necessary functions for changing the rounding modes, and for safe input/output operations.

9.2 Experimental results

BelSyste is implemented in C++ on a machine with the following specifications:

Operating system: Mandrake Linux 9.1.

CPU: Pentium II 450 MHz overclocked to 533 MHz.

Memory size: 128 MB.

Compiler version: gcc 3.2.2.

All the results are obtained using double precision floating-point numbers. To give the reader an idea about the implementation of BelSyste, I included few C++ functions from the system in the appendix.

9.2.1 An application using squashing and propagation

This example is taken from [27], which states that it is a neurophysiology model. It consists

of a system of equations, three of them are inequalities, with variables. The system is

¾ ¾

·  ½¼

¾ ¾

·  ½¼

¿ ¿

·  ¼

¿ ¿

·  ¼

¾ ¾

·  ¿¼

(9.1)

¾ ¾

·  ¾¼



 ¼¼

 ¼¼

½¼¼ ½¼¼ The variables , , , , , and are all in . The input to BelSyste is 9.2 Experimental results 122 precision: 1e-10; Variable : x in [-10.0,10.0]; y in [-10.0,10.0]; z in [-10.0,10.0]; t in [-10.0,10.0]; r in [-10.0,10.0];s in [-10.0,10.0]; Body : solve system xˆ2 + zˆ2 = 1.0; yˆ2 + tˆ2 = 1.0; r*zˆ3 + s*tˆ3 = 5.0; r*xˆ3 + s*yˆ3 = 4.0; r*x*zˆ2 + s*y*tˆ2 = 3.0; r*z*xˆ2 + s*t*yˆ2 = 2.0; y<=x; x >= 0.0; y >= 0.0; display x;y;z;t;r;s;

The first equations are modeled as TreeConstraints. The last three are modeled using the constraint Leq. The whole system is modeled as an ICSystem. When our interval constraint solver uses propagation only, BelSyste’s output is

Duration is: 3 seconds Number of Splits: 103 The number of possible solutions is: 1 Possible Solution 1: x in [0.99722827985234486 , 0.99722827985379403] y in [0.51252181904855099 , 0.51252181905302197] z in [0.074402673741225084 , 0.074402673760646645] t in [0.85867420189183596 , 0.8586742018945045] r in [2.9616005613143641 , 2.9616005613650818] 9.2 Experimental results 123

Strategy Duration Number of splits Propagation 3 103 Box consistency (precision = 0.001) 141 154 Mixed (precision=0.01) 6 23 Mixed (Precision=0.0001) 40 19

Table 9.1. Propagating and squashing in an interval constraint solver. The unit of Dura- tion is seconds.

s in [7.8954846723887817 , 7.8954846724606265]

When the solver uses box consistency (BC) algorithm only, the performance decreases as shown in Table 9.1. But when we use our strategy: use box consistency when domains are

large and propagation when they are small, we get much better performances (the number ¾¿ of splits is reduced from ½¼¿ to ). To minimize the number of splits, we could increase the precision used in the box consistency algorithm. But the running time gets increased.

It is worth mentioning that according to [27], Numerica made ¾ splits to solve the same problem.

9.2.2 Box consistency

To illustrate the performance of BC, ABC, and DBC, we took two examples: one with a single solution, and the other with several solutions.

The first example is to find the intersection between a circle and a parabola:

¾ ¾

·  ½¼

¾



¼¼ ¾¼ where and are both in . 9.2 Experimental results 124

Example Algorithm Duration Number of splits

¼ ¿

BC 

¾½ ¼

First ABC ¾¾

¼ ¼

DBC ¿¼

¼ 

BC ½

 ½  µ ½½ ¾¿ 

ABC ´

 ¾  µ ½ ¿¿½ ½¾¼

ABC ´

 ½  ½ µ  ¼ ¼ ABC ´

Second

 ¾  ½ µ  ¿ ½½

ABC ´

 ½  ½¾µ ¾½ 

ABC ´

 ¾  ½¾µ ½  ½¼

ABC ´

¾½¿ ¾ DBC ½½¼

Table 9.2. Comparison between BC, ABC, and DBC. The unit of Duration is microsec- onds.

The second example is to find the union of two circles:

¾ ¾ ¾ ¾

´´ ½¼µ · ¼¾µ´´ ¿¼µ · ¼¾µ  ¼¼

½¼¼¼ ½¼¼¼ where and are in . The comparison between the algorithms BC, ABC, and DBC, was done based on their durations and the number of splits. The number of splits measures the number of unsuc-

cessful attempts the algorithm did before terminating. Table 9.2 shows the results obtained.

Note how the performance of ABC depends on the parameters and . The parameter de-

termines the width of the interval to try next (more precisely, the width is given by ¾¼¼ ), and is the value to multiply with when a boundary interval is removed. Note that for the first example ABC made no split at all because ABC takes full advantage of the im-

provements suggested in Chapter . When ABC removes a boundary interval, propagation 9.2 Experimental results 125

is applied to reduce the other intervals. The benefit of doing so is obvious in this case.

9.2.3 Box satisfaction

To test box satisfaction algorithms, let us consider the following ICSP

¾ ¾

· ½¼ ½¼¼ ½¼¼ ½¼¼ ½¼¼

(9.2)

½ ½ By running propagation algorithm, the intervals of and get reduced to . Using the zoomin algorithm, we obtain the following output:

Duration : 10326 microseconds Number of splits : 38 Area : 1.733 x = [-0.865234375 , 0.865234375]

y = [-0.5009765625 , 0.5009765625]

¼ ¼¼µ Starting from the point ´¼ , the zoomout algorithm gives

Duration : 4315 microseconds Number of splits : 20 Area : 0 x = [-1 , 1] y=[0,0]

Because the zoomout algorithm works on one variable at a time, it stretched the domain of

to its maximum value. So, there is no room for stretching the domain of . To obtain better results, we used the dynamic algorithm zoomoutbl. This algorithm stretches the domains in a cyclic manner. The output of the zoomoutbl algorithm is given below.

Duration : 40202 microseconds Number of splits : 168 Area : 1.861 x = [-0.82604852928488981 , 0.82604852928488981] y = [-0.56340753044452541 , 0.56340753044452541] 9.2 Experimental results 126

9.2.4 Box unsatisfaction

Let us now consider the following ICSP:

¾ ¾

· ½¼ ½¼¼ ½¼¼ ½¼¼ ½¼¼ (9.3)

The algorithm zoomin for box unsatisfaction returned the following results:

Duration : 16688 microseconds Number of splits : 64 Area : 1.951 x = [-0.78006744384765625 , 0.780029296875]

y = [-0.62557220458984375 , 0.6256103515625]

¼ ¼¼µ Starting from the point ´¼ , the output of the zoomout algorithm is:

Duration : 13209 microseconds Number of splits : 56 Area : 0.087 x = [-0.999755859375 , 0.999755859375] y = [-0.02197265625 , 0.02197265625]

The dynamic algorithm zoomoutbl allowed us to obtain wider rectangle as shown below.

Duration : 75675 microseconds Number of splits : 352 Volume : 1.928 x = [-0.794531113034051 , 0.794531113034051] y = [-0.60693546781760932 , 0.60693546781760921]

9.2.5 Largest box in a convex set

A problem that arose when we dealt with box satisfaction is: given an ICSP and a point

in the solution set of , what is the largest box of center on which is box satisfied?

Ò

Suppose that the box is in a space of dimension , and ½ are the variables in

´¼ ¼µ . Without loss of generality, suppose also that is . The problem can then be 9.2 Experimental results 127

formulated as the constrained global optimization problem in which we want to maximize

´ µ 

Ò ½ ¾ Ò

½ , subject to . The problem is thus equivalent to

¾ Ò ½ (9.4)

As an illustration, let us suppose that is the ICSP

¾ ¾

· ½¼ ¼¼ ½¼ ¼¼ ½¼

½ ¾

½ ¾

The global optimization problem (9.4) becomes

¾ ¾

· ½¼ ¼¼ ½¼ ¼¼ ½¼

¾ ½ ¾

½ (9.5)

½ ¾

The problem (9.4) is equivalent to the unconstrained global optimization problem

Õ

¾

½ ¼¼ ½¼ ½

½ (9.6)

½

Ô

¾

´ µ  ½ ½

Using the plotting capability of BelSyste, the graph of the function ½

½

¼¼ ½¼ ¼¼ ½¼

with is shown in Figure 9.3. The -axis is from to , and the -axis is

½¼ ¼¼ from to . As this figure shows, the function is a convex function with one global

minimum.

¿ With a precision of ½¼ and using our interval constraint algorithm for unconstrained global optimization with the usual splitting strategy, we obtain the following output: fmin is in [-0.50004619219133384, -0.49999999649275495] The number of boxes on which f attains the lower bound is: 1 Duration is: 1 second Number of splits is: 8 Number of boxes is: 2 Number of atomic boxes is: 2 Volume of the cover: 0.00013

When our splitting strategy, which is: split the largest interval near its midpoint but far from the point of exploration, is used, we obtain

9.2 Experimental results 128

¾

 ½ ¼¼ ½¼ ½¼ ¼¼ Figure 9.3. The graph of , where and in , using BelSyste. 9.2 Experimental results 129 fmin is in [-0.50004619219133384, -0.49999999649275495] The number of boxes on which f attains the lower bound is: 1 Duration is: 0 second Number of splits is: 8 Number of boxes is: 2 Number of atomic boxes is: 2 Volume is: 0.00013

These results were obtained using the default explorer: use the midpoint to update the upper bound for the global minimum. When a better explorer, such as the explorer with propagation (explained in the six-hump function example), is used, the number of splits

drops to ¿ as show below. fmin is in [-0.500055540992974, -0.49999999999999989] The number of boxes on which f attains the lower bound is: 1 Duration is: 0 second Number of splits is: 3 Number of boxes is: 3 Number of atomic boxes is: 3 Volume is: 9.473e-05

The benefit of our splitting strategy is more apparent in the case of constrained global optimization. In fact, with the usual splitting strategy, our interval constraint algorithm for constrained global optimization gave the following results. fmin is in [-0.50006090433480832, -0.49998323450439131] The number of boxes on which f attains the lower bound is: 1 Duration is: 2 seconds Number of splits is: 371 Number of boxes is: 208 Number of atomic is: 208 Volume is: 8.236e-07 9.2 Experimental results 130

Using our splitting strategy, we obtain

fmin is in [-0.50006843878498053, -0.49999133224546882] The number of boxes on which f attains the lower bound is: 1 Duration is: 1 second Number of splits is: 336 Number of boxes is: 180 Number of atomic boxes is: 180 Volume is: 5.954e-07

If we use the explorer with propagation, we get much better results:

fmin is in [-0.50006209861466455, -0.49998413446810691] The number of boxes on which f attains the lower bound is: 1 Duration is: 1 second Number of splits is: 283 Number of boxes is: 158 Number of atomic boxes is: 158 Volume is: 6.632e-07

The power of interval constraints over interval arithmetic can easily be seen in the ½ results obtained from the constrained version of MS¾ and from FPFS . The results of the

constrained version of MS¾ are: fmin is in [-0.50006581842899323, -0.49997949972748756] The number of boxes on which F attains the lower bound is: 1 Duration is: 1 second Number of splits is: 1048 Number of boxes is: 255 Number of atomic boxes is: 76 Volume is: 0.75001848489046097

The results obtained from FPFS½ are: 9.2 Experimental results 131

fmin is in [-0.50006581842899323, -0.49993634223937988] The number of boxes on which f attains the lower bound is: 1 Duration is: 2 seconds Number of splits is: 2078 Number of boxes is: 628 Number of atomic boxes is: 628 Volume is: 2.339482307434082e-06

9.2.6 Booth function

To illustrate the power of exploration, let us consider the Booth function given below. It is

 ¼

is a 2-variable function that has several local minima with the global minimum Ñ
Ò at

 ½  ¿

:

¾ ¾

´ µ  ´ · ¾ µ · ´¾ · µ  ·  · with

Without exploration, we obtain the following results,

Duration is: 723668 microseconds fmin is in [0, 1.4482779898067875e-07] The number of boxes on which f attains the lower bound is: 6 Boxes on which f attains the lower bound are: x=[0.99980904413360228, 1.0003805624771058] y=[2.9996194375228939, 3.0000013492556898] x=[0.99961943752287841, 1.0001896066107079] y=[3.0000013492556898, 3.0003805624771376] x=[-1.7976931348623157e+308, -1.7976931348623155e+308] y=[1.7976931348623157e+308, inf] x=[1.7976931348623157e+308, inf] y=[-1.7976931348623157e+308, -1.7976931348623155e+308] x=[1.7976931348623157e+308, inf] y=[-inf, -1.7976931348623157e+308] 9.2 Experimental results 132

x=[-inf, -1.7976931348623157e+308] y=[1.7976931348623157e+308, inf] Number of splits is: 124 Number of boxes is: 13 Number of atomic boxes is: 13 Volume in is: inf

Using the Matlab function fminsearch, which uses a direct search method, with an initial

µ  ´¼¼ ¼¼µ  ´½¼¼¼¼ ¿¼¼¼¼µ

point ´ as our explorer, we obtain the exploration point



with Ü 7.5972e-009. In this case, BelSyste gives the following results:

Duration is: 470875 microseconds fmin is in [0, 0] The number of boxes on which f attains the lower bound is: 5 Boxes on which f attains the lower bound are: x=[0.99999999999999978, 1.0000000000000002] y=[2.9999999999999996, 3.0000000000000004] x=[1.7976931348623157e+308, inf] y=[-1.7976931348623157e+308, -1.7976931348623155e+308] x=[1.7976931348623157e+308, inf] y=[-inf, -1.7976931348623157e+308] x=[-1.7976931348623157e+308, -1.7976931348623155e+308] y=[1.7976931348623157e+308, inf] x=[-inf, -1.7976931348623157e+308] y=[1.7976931348623157e+308, inf] Number of splits is: 26 Number of boxes is: 5 Number of atomic boxes is: 5 Volume is: inf 9.2 Experimental results 133

9.2.7 Rosenbrock banana function

The Rosenbrock banana function is a 2-variable function given as

¾ ¾ ¾

´ µ  ½¼¼´ µ · ´ ½µ  ·  ·

with

 ¼¼ ´½ ½µ

It has a global minimum Ñ
Ò that occurs at . Without exploration, BelSyste gives the following results: fmin is in [0, 2.6976043000745695e-06] Duration is: 496781 microseconds The number of boxes on which f attains the lower bound is: 2 Boxes on which f attains the lower bound are: x=[0.99990091904048239, 1.000626833692323] y=[1.000164794921875, 1.0008911132812499] x=[0.99953765788919946, 1.0002638361681127] y=[0.99943847656249996, 1.000164794921875] Number of splits is: 37 Number of boxes is: 11 Number of atomic boxes is: 11

Volume is: 3.5578688655168111e-06

µ  ´¼¼ ¼¼µ

Using the function fminsearch from Matlab with an initial point ´ as our

 ´½¼¼¼¼ ½¼¼¼¼µ  explorer, we obtain the exploration point with Ü 3.6862e-010. This explains the results shown below. fmin is in [0, 0] Duration is: 394460 microseconds The number of boxes on which f attains the lower bound is: 1 Boxes on which f attains the lower bound are: x=[1 , 1] y=[1 , 1] Number of splits is: 0 9.2 Experimental results 134

Number of boxes is: 1 Number of atomic boxes is: 1 Volume is: 0

Note that no splitting is needed when exploration is used. This example, as well as the previous one, illustrates also the power of our splitting strategy. The box containing the minimizer was not bisected. This gives less boxes on

which attains the lower bound.

9.2.8 Six-hump camel function

Six-hump function is a 2-variable function with six local minima, two of them are global.

The function is given by

¾  ¾ ¾ ¾

´ µ  ´ ¾½ · ¿µ · ·´· µ ¿¼ ¿¼ ¾¼ ¾¼

with

¿   Table 9.3 gives the performance of the algorithms MS¾ , MS , MS , and MS . In the table, we used the following abbreviations:

Algo: Algorithm.

Prec: Precision.

D: Duration.

NbrSp: Number of Splits.

NbrAt: Number of atomic boxes in the cover.

Tt: Number of boxes in the cover. NbrAtn: the number of atomic boxes on which attain the lower bound.

Note that box precision, range precision, inner approximation precision, and outer approx-

imation precision are set to the same value given by Prec.



Using a precision of ½¼ , the atomic boxes on which attains the lower bound are

¼ ½ ¼¼ ¼ ¼½¾ ¼½¾  ¼¼ ¼ ¼¼ ½ ¼½¾ ¼½¾

¼ and .

¼¼

Note that the volumes in MS are because the inner approximations contain degenerate boxes only.

9.2 Experimental results 135

Algo Prec Interval for Ñ
Ò D NbrSp NbrAt/Tt Volume NbrAtn

½

½¼ ¾¿ ¼  ¼ ¼ ¾½  ½¿¾ ¾

¾

½¼ ½¼ ½¼¿½ ¼ ½ ½ ½¼¼ ¾¾ ¼  ¾

MS ¾

¿

½¼ ½¼¿ ½¼¿½  ¾  ½ ½¾ ¾¾ ¼¼¼ ¾



½¼ ½¼¿½ ½¼¿½ ¾ ¼   ¿ ¿¼ ¿ ¾¿¼ ¼¼¼ ¾

½

½¼ ¾¿ ½¼¾ ¼  ¾¾¾¾ ½¿¿ ¾

¾

½¼ ½¼ ½¼¿½ ½ ¾    ¼¼¿¼½ ¾

MS ¿

¿

½¼ ½¼¿ ½¼¿½   ½¾   ¾¾ ¾¾ ¼¼¼¿ ¾



½¼ ½¼¿½ ½¼¿½ ¾ ¿¿ ½¿¿ ¿  ¿  ¿ ¼¼¼¼¾ ¾

½

½¼ ¾¿ ¼ ¾ ¼ ¿½ ¾¾ ½½½ ¾

¾

½¼ ½¼ ½¼¾½ ½ ¾  ¿ ¼ ¼ ¼¼½ ¾

MS 

¿

½¼ ½¼¿ ½¼¿¼  ½¼ ½ ¼½¿   ¼¼¼ ¾



½¼ ½¼¿½ ½¼¿½¾  ½ ½¿ ¾   ¿  ¿ ¼¼¼¼¿ ¾

½

½¼ ¾¿ ¼ ¾ ¼ ¿½ ¾¼¾¼ ¼¼

¾

½¼ ½¼ ½¼¾½ ¾ ¾  ¿ ¾¾ ¼¼

MS 

¿

½¼ ½¼¿ ½¼¿¼  ½ ½ ¼½¿ ½ ½  ¼¼



½¼ ½¼¿½ ½¼¿½¾ ½ ½ ½¿ ¾  ½ ½  ¼¼

¿   Table 9.3. Performance of MS ¾ ,MS ,MS , and MS . The unit of D is seconds.

9.2 Experimental results 136

¿¼ ¿¼

It is worth pointing out that the interval arithmetic evaluation of on the box

¾¼ ¾¼  ½ ½ ¿¿¿

gave . The first bisections improved the lower bound only by

½ ¾¾¼ ½ ¾ 

. Using interval constraints, the interval was reduced to .



With a precision of ½¼ , our interval constraint solver for global optimization gave the following results, which were obtained without exploration. fmin is in [-1.0319645585096058, -1.0316284529256954] Duration is: 500 seconds Number of splits is: 28387 Number of boxes is: 11939 Number of atomic boxes is: 11939 Volume is: 5.4083051386472114e-05

Using exploration, we obtained the results below. The explorer used is called explorer

with propagation, and is as follows. Given a box , we apply the propagation algorithm

on the ICSP

¾

 ·  ´½ · ¾ µ  ¼ ´ µ

¼ ¼

When propagation terminates, we obtain a box . The midpoint of is then used

as our exploration point . The reasons for using only one component of the gradient of ,

´ µ denoted by , are:

to illustrate the fact that exploration is used to extract partial information and not to

´ µ  ´¼ ¼µ solve , and

to speed up propagation by using the component with the simpler expression.

fmin is in [-1.0319645585096058, -1.0316284533220543] Duration is: 480 seconds Number of splits is: 25698 Number of boxes is: 10249 Number of atomic boxes is: 10249 Volume is: 5.5169521459283468e-05 9.2 Experimental results 137

Algorithm Duration (us) Number of splits Lower bound

¾¿ ½  ¾¼

squashLftC ½½¾

 ½¼¿½  ½½¾µ ¾¿ ¾  ¾½¼

ExplLB ´

 ½¼¿½  ¾¼¼µ  ¼ ½½  ¾¼ ExplLB ´

Table 9.4. Comparing ExplLB with squashLftC.

Six-hump function is a good candidate to compare squashLftC given in Chapter  with

¿ ½¼ ExplLB given in Chapter . The precision used in both algorithms is . Table 9.4 shows the results.

9.2.9 Discrete examples

The purpose of this examples is to illustrate the use of the unary constraint int to solve discrete problems.

9.2.10 An inequality on integers

¿

¼

Let us consider the following inequality taken from [5]: , where and are

½¼¼ in ½ . Our goal is to find all the values of and that satisfy this inequality.

First, we consider the inequality on reals, and decompose it into a set of primitive

¿

   · ¼

constraints as follows: . Next, we add to this set, the constraints stating that the variables , and are integers. Thus, our interval constraint

system is:

¿

   · ¼ ´µ ´ µ

int int (9.7)

½ ½¼¼  ·

where , are in , and , , and are in .

½ ¿

Applying GPA to ICSP (9.7) reduces the domain of to and the domain of to

¿ ½ . Since these intervals are still large, we need to split the ICSP. If we choose to bisect the largest interval first, then we get a list of point-intervals satisfying the inequality. Such

a list can be made shorter if we choose to bisect only. The result outputted by our interval constraint solver is as follows. 9.2 Experimental results 138

Duration is: 0 second Number of Splits: 3 The number of possible solutions is: 3 Possible Solution 1: [Safe] xin[3,3] yin[1,1] Possible Solution 2: [Safe] xin[2,2] yin[1,5] Possible Solution 3: [Safe] xin[1,1] yin[1,41]

Note the word “Safe” next to the possible solutions. This means that the interval constraint solver was able to prove that all the values in the intervals returned are solutions to ICSP (9.7). Note also that only three splits were sufficient to find all the solutions.

9.2.11 Integral points on elliptic curves

This example is taken from [55]. The goal is to find the integral points on the elliptic curve

¾ ¿

 ½ ½ ½¼¼¼ given by , where and are in .

The constraint system in this case is

¾ ¿

´ ½µ  ¼ ´µ ´ µ

int int

½ ½¼¼¼ where and are in . Duration is: 1 second Number of Splits: 10 The number of possible solutions is: 1 Possible Solution 1: [Safe] xin[5,5] yin[9,9] 9.2 Experimental results 139

9.2.11.1 Fermat’s last theorem

The famous Fermat’s last theorem states that there is no solution to the equation

Ò Ò Ò

· 

¾

where , , and are integers, and is an integer greater than .

 ¼ ½ ½¼¼ ½ ½¼¼¼

Let us consider and take and in , and in . To break the

symmetry between and we impose the order . Therefore, our interval constraint

system is

¼ ¼ ¼

·  ´µ ´ µ ´ µ

int int int (9.8)

½ ½¼¼ ½ ½¼¼¼ where , and are in , and is in .

Any solver using integer arithmetic with  bits will give rise to overflow when the value

of one of the variables is greater than ½. With our interval constraint solver the overflow is

avoided thanks to extended real-arithmetic and interval arithmetic. However, due to the loss

of accuracy near · , our solver outputted a list of possible solutions containing solutions

½ ½    

such as , and . These solutions can be avoided by adding the

· ½ redundant constraint to ICSP (9.8). With this constraint, our solver was able to prove the absence of solutions without any split and in less than seven seconds.

9.2.11.2 Taxicab number

The th taxicab number is the smallest integer that can be written as the sum of two cubes

in different ways. The first nontrivial taxicab number is known as the Hardy-Ramanujan

number, and is equal to ½¾ . The story associated with this number can be found in [26,

25]. The reader may refer to [65] for a short tutorial on taxicab numbers.

´µ

In what follows we denote by the th taxicab number. Let us show how to model

´¾µ 

as an interval constraint system. ¾

The first constraint is that should be written as the sum of two cubes in ways.

¿ ¿ ¿ ¿

 ·  ·

Thus, we have and , for some integers , , , and . To break the

symmetry of and , and and , we add the constraints and to our constraint

9.2 Experimental results 140

µ ´ µ

system. The symmetry between ´ and can be eliminated by the constraint .

Because the two ways of writing have to be different, this constraint becomes , that

· ½

is ; Thus, our interval constraint system is

¿ ¿ ¿ ¿

 ·  · · ½ ´µ

int

µ ´ µ ´µ ´ µ

int´ int int int

½ ½¼¼¼ ½ ·

where , , , and are all in , and in Using a depth-first search, our

 ½¾  ½  ½¾

solver returned the first number in less than one second, with , ,

  ½¼ , and .

9.2.12 BelSyste as a verification tool

From the paper [48], which reported the experimental results obtained from testing most of global optimization solvers, we have the following remarks:

None of the global optimization solvers tested was fully reliable.

BARON, which is a global optimization solver, was the fastest but missed the global

minimum in about ± of the cases.

ICOS was the most reliable one but handles only pure constraint satisfaction prob- lems, and was very slow compared to BARON and LINGO.

From these remarks, it is obvious that there is a need for a global optimization solver that can verify fast solvers such as BARON [52]. To illustrate this, let us consider the following

example taken from [36]:

´ µ

where is given by the ICSP

¾

 ¼

¾ 

´ ½µ · ½¼  ¼

½¼¼ ½¼¼

9.2 Experimental results 141

¾ ¾ 

 ¼ ´ ½µ · ½¼  ¼ ¼ ¼

Figure 9.4. The graph of and , where

½ ¼ ½ ¼

and in , using BelSyste.

¼ ´¼¼ ¼¼µ

For this example, BARON 7.2 returned ¼ for the global minimum and for

¼½

the minimizer. Using a precision of ¼ , and running propagation on the ICSP above with

¼¼½

, BelSyste showed that the domain of is empty in less than one second. Therefore

¾ ¾ 

 ¼ ´ ½µ · ½¼  ¼

BARON 7.2 results are not correct. The graphs of and

  ½ ½  with and are shown in Figure 9.4. Although the graphs appear to interset at two points, they only intersect at one point

with a positive . This explains why BARON 7.2 gives the wrong global minimum. The

graph of the region is shown in Figure 9.5 with a gray color. For the example above,

¿¼½ BelSyste returned the interval ¾ for the minimum of in less than one second.

9.2 Experimental results 142

¾ ¾ 

 ¼ ´ ½µ · ½¼  ¼ ¼ ¼

Figure 9.5. A graph for with

½ ¼ ½ ¼ and in using BelSyste. Chapter 10

Conclusion

In this chapter, we briefly discuss our contributions in interval methods for global optimiza- tion. We then suggest topics for future work.

10.1 Contributions

10.1.1 Interval arithmetic

The usual interval arithmetic algorithms for global optimization suffer from at least one of the following drawbacks:

Mix the fathoming and the localization problems. Do not handle the inner and outer approximations for -minimizer.

Have properties that are correct only in the limit for infinite running time, infinite memory, and infinite precision of the floating-point number system.

The algorithms we proposed in the first part of our thesis remove all these drawbacks. Our results are qualitatively different, so time comparison between our algorithms and the existing ones is irrelevant.

10.1.2 Interval constraints

10.1.2.1 Propagation

Propagation is the key algorithm in interval constraints. It is the basic building block of other algorithms such relational box-consistency, interval constraint solving, and interval 10.1 Contributions 144 constraint algorithms for global optimization. All these algorithms call propagation. This explains our efforts to improve the generic propagation algorithm as much as possible. All our algorithms, namely dual, clustered, deterministic, and selective propagation algorithms, were developed to speed up the generic propagation algorithm.

10.1.2.2 Relational box-consistency

Relational box-consistency algorithm is another key algorithm in interval constraints. The algorithm keeps squashing the left and then right of each domain until no pruning is pos- sible. The main drawback with this way of squashing is that as we proceed further, the process of pruning becomes slow. To tackle these problems, we proposed the following algorithms:

1. Dynamic Box-Consistency (DBC) algorithm Instead of pruning the left and then the right of each domain, we alternate the pruning between all the domains.

2. Adaptive Box-Consistency (ABC) algorithm The idea behind ABC is to get rid of the boxes as soon as possible: start with small boxes and extend them or shrink them depending on the pruning outcome. This adaptive behavior makes ABC algorithm very suitable for quick squashing.

10.1.2.3 Interval constraints with exploration

Point methods are fast compared to interval constraints but not reliable. However, they can be used as a starting point to speed up interval computations that would otherwise be very slow. The technique of using point methods to speed up interval constraints is called exploration. So, by combining the efficiency of classical algorithms, and the safety of interval methods, we get the ultimate goal of any numerical analyst: fast computations, and reliable results. In the last chapter of the second part of our thesis, we discussing the effect of explo- ration on

the lower and upper bound of the global minimum, 10.2 Future work 145

the splitting of the search space,

the interval constraint solver, and

the box-consistency algorithms.

We also introduced box satisfaction and box unsatisfaction concepts, and the necessary algorithms for them.

10.1.3 Implementation

In our implementation of interval algorithms for global optimization, we looked for sim- plicity, correctness, and extendibility. To achieve this, we made extensive use of software engineering techniques; design patterns in particular. In the sense of [15], the design pat- terns used include:

Observer: when the domain of a variable changes, we want to notify the constraints in which it is involved. This is the reason for using the Observer design pattern in our implementation.

Strategy: each constraint has its own DRO; this suggest using Strategy pattern to uniformly handle all constraints.

Composite: this design pattern was used to capture the hierarchical structure of the ICSPs.

Decorator: propagation and box-consistency algorithms are examples of consistency algorithms. So, instead of applying propagation on an ICSP, we may want to use other consistency algorithms such as box-consistency algorithms. To achieve this, we used the Decorator design pattern. This helped us to treat uniformly all consistency algorithms.

10.2 Future work

These are some of the main problems I left unsolved during my research in interval meth- ods. 10.2 Future work 146

10.2.1 Dependency effect

It is known that the source of the excessive over estimation in interval methods is the dependency effect. This effect happens when variables have multiple occurrences in an expression. So, the excessive over estimation has to do with the number of occurrences of a variable. But that’s not all. It has to do also with its domain and the domains of the other

variables. As an illustration, let us consider the following expression:

¾

´ µ  ·

¼ · ¼ ·

with . If the interval of is a subset of , the number of occurrences

of does not introduce any over estimation. But when the interval of is a subset of

¼  , the number of occurrences of takes effect. So, the problem is how to quantify the dependency effect of each variable, and how to know when the over estimation is present. Is it possible to know which variable causes the “most” dependency effect? The answer to this question is of great importance to interval methods because of the following observation:

if we could order the variables by their effects, we would use this information to bisect the domain of the variable with the ”most” dependency effect, and thus sharpen the bounds of the interval evaluations.

10.2.2 Parametric CSPs

The CSPs we dealt with so far involve atomic formulas with only constants and variables.

In some fields, such as robust control, atomic formulas can also involve parameters. As an

¾

Ê ½ ¾ ½

example, we may want to find such that we have · . In this

¾

· ½ ½ example, the atomic formula has a variable, namely , a constant, , and a

parameter . In addition to variables, a constraint in a parametric CSP can also have parameters. So, the DRO of a constraint must distinguish between a variable and a parameter. In fact, the domain of a variable may get reduced, but the domain of a parameter must not. If it does, then the parametric CSP considered does not have a solution. Similarly, squashing 10.2 Future work 147

a parameter and a variable must be distinguished. When squashing a parameter, if we succeed to eliminate a part of its domain, then the parametric CSP does not have a solution. In [42], I showed how to implement parametric CSPs how to use them to study the robust stability of a dynamical system. The work done, however, is still incomplete. Among other things, a parametric constraint solver is not yet developed.

10.2.3 Criteria for choosing a consistency algorithm

Given a set of equations, consistency algorithms, including propagation and squashing al- gorithms, are used to reduce the domains of the variables to a fixpoint. Although two con- sistency algorithms may give the same fixpoint, their efficiencies can differ a great deal. So, given a set of equations, is there any way we can know in advance the most efficient consistency algorithm to use? Even when a consistency algorithm has a better fixpoint than another, it may happen that, in the long run, the latter algorithm may be more efficient and output the same result as the former. As an illustration, in the interval constraint solver and the interval constraint algorithm for global optimization, we left the choice of the con- sistency algorithm used to the user. That is, he can either use a squashing algorithm or a propagation algorithm. Our heuristics of using squashing when the domains are big and propagation when they are small does not have any theoretical foundation; it is merely an observation. Therefore, it would be nice to have well-founded criteria for choosing the best consis- tency algorithm.

10.2.4 Confidence quantification

To know how far from the upper bound of Ñ
Ò we should split, we introduced the con- fidence concept. We mentioned that a way to improve our confidence in the result of an exploration, is to do more explorations. If all the explorers land at the same point, then our confidence about the results should be increased. This arises several questions such as: how to quantify this confidence? How many explorations are necessary in each exploration step? Do we need to use different algorithms in the exploration step? 10.2 Future work 148

10.2.5 Disjunctive constraints

The CSPs that we dealt are conjunctive in the sense that they involve atomic formulas with

the “and” operator between them. There are many numerical problems that involve atomic formulas related with both “and” (denoted by ) and “or” (denoted by ) operators. As a

simple example, solve for the following system:

¾ ¿

 ´ · ½  ¼ · ½µ

This kind of CSPs is “disjunctive” since it involves both operators. As opposed to con- junctive CSPs that involve intersections only (that’s why they are easier), disjunctive CSPs involve intersections and unions.

10.2.6 Iterative widening with exploration

Numerica [27] introduced a novel technique, called iterative widening, to enlarge the set

´ µ Ò of solutions of an ICSP. Starting from a point ½ near a solution, produced by

a point method for example, iterative widening tries to find other solutions in the boxes

  ·   ·   ½ ¾

½ ½ Ò Ò
for , using interval methods

or continuation methods [47, 64]. A drawback of this technique is that it does not find all

´ µ Ò solutions. Moreover, when the point ½ is not very close to a solution, iterative widening becomes very slow. We think that the way in which we used exploration to improve box-consistency algorithms can also help iterative widening to efficiently tackle these drawbacks. Bibliography

[1] “IEEE standard for binary floating-point arithmetic,” ANSI/IEEE, Tech. Rep. Stan- dard 754-1985, 1985. [2] G. Alefeld and J. Herzberger, Introduction to Interval Computations. Academic Press, 1983. [3] K. Apt, “The essence of constraint propagation,” Theoretical Computer Science, vol. 221(1-2), pp. 179–210, 1999. [4] K. R. Apt, Principles of Constraint Programming. Cambridge University Press, 2003. [5] K. R. Apt and P. Zoeteweij, “A comparative study of arith- metic constraints on integer intervals,” 2004. [Online]. Available: http://www.citebase.org/abstract?id=oai:arXiv.org:cs/0403016 [6] R. Bartak, “Theory and practice of constraint propagation,” in Proceedings of the 3rd Workshop on Constraint Programming in Decision and Control (CPDC 2001), J. Figwer, Ed., 2001, pp. 7–14. [7] F. Benhamou, D. McAllester, and P. V. Hentenryck, “CLP(Intervals) revisited,” in Logic Programming: Proc. 1994 International Symposium, 1994, pp. 124–138. [8] F. Benhamou, F. Goualard, L. Granvilliers, and J.-F. Puget, “Revising hull and box consistency,” in Proceedings of the 16th International Conference on Logic Program- ming. MIT Press, 1999, pp. 230–244. [9] H. Chen, “Global optimization using interval constraints,” Ph.D. dissertation, Univer- sity of Victoria, 1998. [10] R. Dechter, Constraint Processing. Morgan Kaufmann, 2003. [11] R. P. Feynman, R. B. Leighton, and M. Sands, Feynman«s lectures on physics, Vol. 2. Addison-Wesley, 1964. [12] R. Fletcher, Practical Methods of Optimization. Wiley, 1987. [13] R. Fletcher, “An overview of unconstrained optimization,” 1994. [Online]. Available: citeseer.ist.psu.edu/fletcher93overview.html Bibliography 150

[14] R. Fourer, D. M. Gay, and B. Kernighan, AMPL: A Modeling Language for Mathe- matical Programming, 2nd ed. Duxbury Press, 2003. [15] E. Gamma, R. Helm, R. Johnson, and J. Vlissides, Design Patterns. Addison-Wesley, 1994. [16] F. Glover and M. Laguna, “Tabu search,” in Modern Heuristic Techniques for Combinatorial Problems, C. Reeves, Ed. Oxford, England: Blackwell Scientific Publishing, 1993. [Online]. Available: citeseer.ist.psu.edu/glover97tabu.html [17] D. E. Goldberg, Genetic Algorithms in Search, Optimization, and Machine Learning. Addison-Wesley Professional, January 1989. [18] L. Granvilliers, F. Goualard, and F. Benhamou, “Box consistency through weak box consistency,” in ICTAI, 1999, pp. 373–380. [Online]. Available: citeseer.ist.psu.edu/granvilliers99box.html [19] E. R. Hansen and G. Walster, “Nonlinear equations and optimization,” Preprint, 1987. [20] E. Hansen, Topics in Interval Analysis. Oxford University Press, 1969. [21] E. Hansen, “Global optimization using interval analysis – the one-dimensional case,” Jour. Optimiz. Theory Applic., vol. 29, pp. 331–344, 1979. [22] E. Hansen, “Global optimization using interval analysis – the multi-dimensional case,” Numer. Math., vol. 34, pp. 247–270, 1980. [23] E. Hansen, Global Optimization Using Interval Analysis. Marcel Dekker, 1992. [24] E. Hansen and S. Sengupta, “Global constrained optimization using interval analysis,” Nickel, pp. 25–47, 1980. [25] G. H. Hardy, A Mathematician’s Apology, with a foreword by C. P. Snow. Cambridge University Press, 1993. [26] G. H. Hardy, Ramanujan: Twelve Lectures on Subjects Suggested by His Life and Work, 2nd ed. American Mathematical Society, 1999. [27] P. V. Hentenryck, L. Michel, and Y. Deville, Numerica: A Modeling Language for Global Optimization. MIT Press, 1997. [28] T. Hickey, Q. Ju, and M. van Emden, “Interval arithmetic: from principles to imple- mentation,” Journal of the ACM, vol. 48, no. 5, 2001. [29] T. J. Hickey, Z. Qiu, and M. H. van Emden, “Interval constraint plotting for interactive visual exploration of implicitly defined relations,” Reliable Computing, vol. 6, pp. 81– 92, 2000. [30] K. Ichida and Y. Fujii, “An interval arithmetic method for global optimization,” Com- puting, vol. 23, no. 1, pp. 85–97, Feb. 1979. Bibliography 151

[31] IC.PARC, “The eclipse constraint logic programming system,” 2003, research Project at Imperial College London. [32] L. Jaulin, M. Kieffer, O. Didrit, and E. Walter, Applied Interval Analysis. Springer- Verlag, 2001. [33] A. H. G. R. Kan and G. T. Timmer, “Stochastic global optimization methods. part 1: clustering methods,” Math. Program., vol. 39, no. 1, pp. 27–56, 1987. [34] S. Kirkpatrick, C. D. Gelatt, and M. P. Vecchi, “Optimization by simulated annealing,” Science, Number 4598, 13 May 1983, vol. 220, 4598, pp. 671–680, 1983. [Online]. Available: citeseer.ist.psu.edu/kirkpatrick83optimization.html [35] I. Lakatos, Proofs and Refutations. Cambridge University Press, 1976. [36] Y. Lebbah, C. Michel, and M. Rueher, “An efficient and safe framework for solving optimization problems,” J. Comput. Appl. Math., vol. 199, no. 2, pp. 372–377, 2007. [37] P. Leclerc, “Efficient and reliable global optimization,” Ph.D. dissertation, The Ohio State University, 1992. [38] Z. Michalewicz, Genetic algorithms + data structures = evolution programs (3rd ed.). London, UK: Springer-Verlag, 1996. [39] K. Miettinen, M. M. Makela, and H. Maaranen, “Efficient hybrid methods for global continuous optimization based on simulated annealing,” Comput. Oper. Res., vol. 33, no. 4, pp. 1102–1116, 2006. [40] B. Moa, “Interval constraints for computer graphics: Plots, tessellations, and an- imations,” in CSP Techniques with Immediate Applications (CSPIA), R. Bartak, M. Silaghi, and M. Zanker, Eds., 2004. [41] B. Moa, “Constraint programming tools,” CO-OP report, 2004. [42] B. Moa, “Interval constraints for flight-envelope clearance,” CO-OP report, 2006. [43] J. Mockus, Bayesian approach to global optimization. New York: Kluwer Academic Publishers, 1989. [44] E. Monfroy and J.-H. Rety, “Chaotic iteration for distributed constraint propagation,” in Selected Areas in Cryptography, 1999, pp. 19–24. [Online]. Available: citeseer.ist.psu.edu/monfroy99chaotic.html

[45] R. E. Moore, “On computing the range of a rational function of variables over a bounded region,” Computing, vol. 15, pp. 1–15, 1976. [46] R. E. Moore, Interval Analysis. Prentice-Hall, 1966. [47] A. Morgan, Solving Polynomial Systems Using Continuation for Scientific and Engi- neering Problems. Prentice-Hall, 1987. Bibliography 152

[48] A. Neumaier, O. Shcherbina, W. Huyer, and T. Vinko, “A comparison of complete global optimization solvers,” 2004. [Online]. Available: URL: http://citeseer.ist.psu.edu/681387.html [49] A. Neumaier, Interval Methods for Systems of Equations. Cambridge University Press, 1990. [50] C. Papadimitriou and K. Stieglitz, Combinatorial Optimization: Algorithms and Com- plexity. Prentice-Hall, 1982. [51] H. Ratschek and J. Rokne, New Computer Methods for Global Optimization. Ellis Horwood/John Wiley, 1988. [52] H. Ryoo and N. Sahinidis, “A branch and reduce approach to global optimization,” J. Global Optim., vol. 8, no. 2, pp. 107–139, 1996. [53] S. Skelboe, “Computation of rational interval functions,” Nordisk Tidskrift for Infor- mationsbehandling (BIT), vol. 14, pp. 87–95, 1974. [54] S. Skelboe, “True worst-case analysis of linear electrical circuits by interval arith- metic,” IEEE Trans. Circuits and Systems, vol. CAS-26, pp. 874–879, 1979. [55] N. Smart, The Algorithmic Resolution of Diophantine Equations. Cambridge Uni- versity Press, 1996. [56] A. Torn and A. Zilinskas, Global Optimization. Springer-Verlag, Berlin Heidelberg, New York, 1989, lecture Notes in Computer Science. [57] E. Tsang, Foundations of Constraint Satisfaction. Academic Press, 1993. [58] J. Tupper, “Reliable two-dimensional graphing methods for mathematicalformulae with two free variables,” in SIGGRAPH 2001, Computer Graphics Proceedings, E. Fiume, Ed. ACM Press / ACM SIGGRAPH, 2001, pp. 77–86. [Online]. Available: citeseer.ist.psu.edu/article/tupper01reliable.html [59] M. van Emden, “Value constraints in the CLP Scheme,” Constraints, vol. 2, pp. 163– 183, 1997. [60] M. van Emden, “Computing functional and relational box consistency by structured propagation in atomic constraint systems,” in Proc. 6th Annual Workshop of the ERCIM Working Group on Constraints; downloadable from CoRR, 2001. [61] M. van Emden and B. Moa, “Propagation by selective initialization and its applica- tion to numerical constraint satisfaction problems,” Department of Computer Science, University of Victoria, Tech. Rep. DCS-286-IR, paper cs.OH/0404034 in Computing Research Repository (CoRR), April 2004. [62] M. van Emden and B. Moa, “Termination criteria in the Moore-Skelboe algorithm Bibliography 153

for global optimization by interval arithmetic,” in Frontiers in Global Optimization, C. Floudas and P. Pardalos, Eds. Kluwer Academic Publishers, 2003. [63] M. van Emden and B. Moa, “The fundamental theorems of interval analysis,” in Scien- tific Computing, Computer Arithmetic and Validated Numerics, W. Luther and W. Ot- ten, Eds. To appear in SCAN06 Proceedings, 2007. [64] J. Verschelde, P. Verlinden, and R. Cools, “Homotopies exploiting newton polytopes for solving sparse polynomial systems,” SIAM J. Numer. Anal., vol. 31, no. 3, pp. 915–930, 1994. [65] E. W. Weisstein, “Taxicab numbers,” 2006. [Online]. Available: From MathWorld, A Wolfram Web Resource, http://mathworld.wolfram.com/TaxicabNumber.html Appendix A

Source Code

A.1 Box consistency //left narrowing to a certain precision prec FLPT BCConstraint::leftNarrowing(Variable* x, const Interval& intr, const double& prec){ ics->activateConstraints(); if (! ics->probe(x,intr)) return intr.getUB(); if (intr.isAtomic(prec)) return intr.getLB(); FLPT m; intr.middlepoint(m);nbreOfSplits++; Interval intL = Interval(intr.getLB(),m); FLPT mres = leftNarrowing(x,intL,prec); if(m==mres){ Interval intR = Interval(m,intr.getUB()); return leftNarrowing(x,intR,prec); } return mres; } //Narrow a variable to precision. //Returns true if changed, false otherwise. bool BCConstraint::narrow(Variable* x){ bool changed = false,lftDone=false,rghtDone=false; while(!lftDone && !rghtDone){ Interval y = x->getValue(); if (!lftDone){ FLPT lb = leftNarrowing(x,y,precision); lftDone = true; if (lb != y.getLB()) { y.setLB(lb); changed = true; rghtDone=false; x->setValue(y); } } if (!rghtDone){ FLPT ub = rightNarrowing(x,y,precision); rghtDone=true; if (ub != y.getUB()) { y.setUB(ub); changed = true; lftDone=false; x->setValue(y); } } A.2 Adaptive box consistency 155

} return changed; } A.2 Adaptive box consistency //left narrowing to a certain precision prec FLPT ABCConstraint::leftNarrowing(Variable* x, const Interval& intr, double step,const double& prec){ FLPT lt=intr.getLB(),ut=intr.getUB(),wt=ut-lt; if (step > wt) step = wt; FLPT l = lt + step; Interval tin(lt,l); //cout << "Left interval tried: "<activateConstraints(); if (!ics->probe(x,tin)) { //cout <getName()<<" left removed = "<setValue(intR); ics->shrinc(); intR = x->getValue(); if (intR.isSingleton()) return intR.getLB(); return leftNarrowing(x,intR,2.0*step,prec); } if (step <= prec) return lt; nbreOfSplits++; FLPT mres = leftNarrowing(x,intr,step/2.0,prec); if(mres==l){ Interval intR(l,ut); return leftNarrowing(x,intR,step,prec); } return mres; } A.3 Dynamic box consistency void DBCConstraint::dbcNarrowing(const vector& unkns, vector& lft, vector& rght, const double& prec) { int size = lft.size(); map lmidpts; for(int i=0; i< size; ++i) { ics->activateConstraints(); if(!ics->probe(unkns[i],lft[i])) { lft[i].setLB(lft[i].getUB()); Interval tval = unkns[i]->getValue(); tval.setLB(lft[i].getUB()); unkns[i]->setValue(tval); //lft[i].setUB(tval.getUB()); }else{ if (!lft[i].isAtomic(prec)) { FLPT m; lft[i].middlepoint(m); nbreOfSplits++; lmidpts[i]=Interval(m,lft[i].getUB()) ; } } } map rmidpts; for(int i=0; i< size; ++i){ ics->activateConstraints(); A.4 Box satisfaction algorithms 156

if(!ics->probe(unkns[i],rght[i])) { rght[i].setUB(rght[i].getLB()); Interval tval = unkns[i]->getValue(); tval.setUB(rght[i].getLB()); unkns[i]->setValue(tval); //rght[i].setLB(tval.getLB()); }else{ if (!rght[i].isAtomic(prec)) { FLPT m; nbreOfSplits++; rght[i].middlepoint(m); rmidpts[i]=Interval(rght[i].getLB(),m); } } } if(rmidpts.size()==0 && lmidpts.size()==0) return; map::iterator it; for(it=lmidpts.begin();it!=lmidpts.end();++it) lft[it->first].setUB(it->second.getLB()); for(it=rmidpts.begin();it!=rmidpts.end();++it) rght[it->first].setLB(it->second.getUB()); dbcNarrowing(unkns,lft,rght,prec); bool again=false; for(it=lmidpts.begin();it!=lmidpts.end();++it) if(lft[it->first].getLB()==it->second.getLB()) { lft[it->first].setUB(it->second.getUB()); again=true; } for(it=rmidpts.begin();it!=rmidpts.end();++it) if(rght[it->first].getUB()==it->second.getUB()) { rght[it->first].setLB(it->second.getLB()); again=true; } if(again) dbcNarrowing(unkns,lft,rght,prec); } A.4 Box satisfaction algorithms //left stretching to a certain precision prec FLPT BSat::stretchLeft(Variable* x, const Interval& r , const double& prec){ bool sat = false; ics->activateConstraints(); ics->probe(x,r,sat); if (sat) return r.getLB();; if (r.isAtomic(prec)) return r.getUB(); FLPT m; r.middlepoint(m); nbreOfSplits++; FLPT res = stretchLeft(x, Interval(m,r.getUB()),prec); if (res == m) return stretchLeft(x, Interval(r.getLB(),m),prec); return res; } //Zoomout to a certain precision void BSat::zoomout(map& frmBx){ ics->storeVariables(); A.4 Box satisfaction algorithms 157

map toBx; map::iterator itr=frmBx.begin(); for(;itr!=frmBx.end();itr++){ toBx[itr->first]=itr->first->getValue(); itr->first->forceValue(itr->second); } FLPT str; for(itr=frmBx.begin();itr!=frmBx.end();itr++){ str=stretchLeft(itr->first, Interval(toBx[itr->first].getLB(), itr->second.getLB()),precision); itr->second.setLB(str); itr->first->forceValue(itr->second); str=stretchRight(itr->first, Interval(itr->second.getUB(), toBx[itr->first].getUB()),precision); itr->second.setUB(str); itr->first->forceValue(itr->second); } ics->restoreVariables(); } //Zoomin to a certain precision void BSat::zoomin(map& frmBx){ ics->storeVariables(); map::iterator itr; set::iterator its; set lm,rm; for(itr=frmBx.begin();itr!=frmBx.end();++itr){ lm.insert(itr->first); rm.insert(itr->first); } while (true) { if (!lm.empty()){ for(its=lm.begin();its!=lm.end();) { Interval tm((*its)->getValue().getLB(), frmBx[*its].getLB()); if (tm.isAtomic(precision)) { lm.erase(its++); }else{ FLPT m; tm.middlepoint(m); tm.setLB(m); (*its)->setValue(tm); ics->activateConstraints(); ics->shrinc(); if (ics->isSatisfied()){ for(itr=frmBx.begin(); itr!=frmBx.end();++itr) itr->second=itr->first->getValue(); ics->attach(); ics->restoreVariables(); zoomout(frmBx); return; } its++; } } } if (!rm.empty()) { for(its=rm.begin();its!=rm.end();) { Interval tm((*its)->getValue().getLB(), frmBx[*its].getLB()); if (tm.isAtomic(precision)) { rm.erase(its++); }else{ FLPT m; tm.middlepoint(m); tm.setLB(m); A.4 Box satisfaction algorithms 158

(*its)->setValue(tm); ics->activateConstraints(); ics->shrinc(); if (ics->isSatisfied()){ for(itr=frmBx.begin(); itr!=frmBx.end();++itr) itr->second=itr->first->getValue(); ics->attach(); ics->restoreVariables(); zoomout(frmBx); return; } its++; } } } if (rm.empty()&&lm.empty()){ ics->restoreVariables(); return; } } } //left stretching to half to a certain precision prec FLPT BSat::stretchLeftToHalf(Variable* x, const Interval& r, const double& prec){ bool sat = false; ics->activateConstraints(); ics->probe(x,r,sat); if (sat){ if (r.isAtomic(prec)) return r.getLB(); FLPT h; r.middlepoint(h); return h; } if (r.isAtomic(prec)) return r.getUB(); FLPT m; r.middlepoint(m); nbreOfSplits++; FLPT res = stretchLeftToHalf(x, Interval(m,r.getUB()),prec); if (res == m) return stretchLeftToHalf(x, Interval(r.getLB(),m),prec); return res; } //Zoomout uniformly to a certain precision void BSat::zoomoutbl(map& frmBx){ ics->storeVariables(); map toBx; map::iterator itr=frmBx.begin(); for(;itr!=frmBx.end();itr++){ toBx[itr->first]=itr->first->getValue(); itr->first->forceValue(itr->second); } FLPT str,ores; bool changed = true; while(changed) { changed = false; for(itr=frmBx.begin();itr!=frmBx.end();itr++){ ores = itr->second.getLB(); str=stretchLeftToHalf(itr->first, Interval(toBx[itr->first].getLB(), itr->second.getLB()),precision); A.5 Default explorer and explorer with propagation 159

if (ores != str){ changed = true; itr->second.setLB(str); itr->first->forceValue(itr->second); } ores = itr->second.getUB(); str=stretchRightToHalf(itr->first, Interval(itr->second.getUB(), toBx[itr->first].getUB()),precision); if (ores != str) { changed = true; itr->second.setUB(str); itr->first->forceValue(itr->second); } } } ics->restoreVariables(); } A.5 Default explorer and explorer with propagation //Default explorer: take the midpoint of a box. Box Explorer::explore(const Box& b, const Interval& rng) { return b.getMiddlepoint(); } //Explorer with propagation. Box PropagationExplorer::explore(const Box& b, const Interval& rng) { Box box = b; Box tmpbx = b; tmpbx.updateProjections(); box.setProjections(); for(unsigned int i=0; igetOutVariable(); out->setValue(Interval(0.0)); derivatives[i]->shrinc(); } box.updateProjections(); tmpbx.setDomains(); return box.getMiddlepoint(); } A.6 Interval constraint global optimization solver with exploration bool ExplConsOptimizer::minimize(const double& rangePrec, const double& boxPrec) { if (constraints == NULL) return uminimize(rangePrec,boxPrec); return cminimize(rangePrec,boxPrec); } bool ExplConsOptimizer::uminimize(const double& rangePrec, const double& boxPrec) { time_t tstart; time(&tstart); Variable* z = objFunction->getOutVariable(); Box box = initBox; ICSystem ics; ics.addConstraint(objFunction); //shrinc all the domains box.setProjections(); objFunction->shrincFst(); BCConstraint bc(&ics,rangePrec); bc.shrinc(z); A.6 Interval constraint global optimization solver with exploration 160

Interval rng = z->getValue(); box.updateProjections(); box.updateAuxiProjections(); BoxesManager atCover;//cover to put small boxes atCover.insert(box,rng); lobound = rng.getLB(); if (upbound>rng.getUB()) upbound = rng.getUB(); Box p; FLPT up,lp,lbnd; bool inbox,isupb; nbreOfSplits=0; while(!atCover.empty()) { inbox = false; isupb = false; pair frstCvr=atCover.top(); box = frstCvr.first; rng = frstCvr.second; atCover.pop(); //exploration step p=explorer->explore(box,rng); box.setAuxiDomains(); p.setProjections(); if(objFunction->shrincFst()){ isupb = true; Interval zp = z->getValue(); up=zp.getUB(); lp=zp.getLB(); p.updateAuxiProjections(); } //reduction step if (isupb && up < upbound){ inbox = true; upbound = up; explPoint = p; box.setExplPoint(p); box.setProjections(); box.setAuxiDomains(); z->setValue(Interval(rng.getLB(),upbound)); if(box.isAtomic(rangePrec)) { bc.shrinc(); objFunction->shrincFst(); }else{ objFunction->shrinc(); } invConfidence = (lp - z->getValue().getLB())/8.0; lbnd=bc.explShrinc(z,z->getValue().getLB(), lp,invConfidence); rng.setLB(lbnd);rng.setUB(upbound); z->setValue(rng); box.updateProjections(); box.updateAuxiProjections(); rng = z->getValue(); }else{ if (rng.getLB()>upbound) continue; box.setProjections(); box.setAuxiDomains(); z->setValue(Interval(rng.getLB(),upbound)); if(box.isAtomic(rangePrec)){ if(!bc.shrinc()) continue; if(!objFunction->shrincFst()) A.6 Interval constraint global optimization solver with exploration 161

continue; }else{ if(!objFunction->shrinc()) continue; } bc.shrinc(z); box.updateProjections(); box.updateAuxiProjections(); if(!objFunction->shrincFst()) continue; } rng = z->getValue(); } if (box.isAtomic(boxPrec)){ cover.insert(box,rng); continue; } //branch-and-bound //Bisect using Belaid’s strategy. box.setProjections(); Variable* var = varChooser->select(boxPrec); FLPT midpoint; Interval intrv = var->getValue(); FLPT varu=intrv.getUB(),varl=intrv.getLB(); FLPT bspVal=box.getExplPoint().getProjection(var).getLB(); //which part from bsp is larger. if (varu-bspVal>bspVal-varl&&bspVal-varl>=0.0){ intrv.setLB(bspVal); }else if (inbox&&bspVal-varl>=varu-bspVal&&varu-bspVal>=0.0){ intrv.setUB(bspVal); } intrv.middlepoint(midpoint); nbreOfSplits++; Box bxl = box; Box bxr = box; bxl.setProjection(var,Interval(varl,midpoint)); bxr.setProjection(var,Interval(midpoint,varu)); atCover.insert(bxl,rng); atCover.insert(bxr,rng); } cover.remove(upbound); lobound = cover.top().second.getLB(); time_t tend; time(&tend); duration = difftime(tend,tstart);//clock()-tstart; return true; } VITA

Surname: Moa Given Names: Belaid Place of Birth: Boumia, Khenifra, Morocco

Educational Institutions Attended

University of Victoria 2002 to 2007

Educational Institutions Attended

Institut National Polytechnique De Toulouse (DEA courses) 1999 to 2000

Educational Institutions Attended

Ecole National Polytechnique Superieure´ ENSEEIHT (M.Eng.) 1998 to 2000

Educational Institutions Attended

Class Preparatoires´ and Ecole Hassania des Travaux Publics (B.Eng.) 1994 to 1998

Degrees Awarded

B.Eng. Ecole Hassania des Travaux Publics (EHTP) 1998 M.Eng. Ecole National Polytechnique Superieure´ ENSEEIHT 2000

Publications

1. M.H. van Emden and B. Moa, “The Fundamental Theorems of Interval Analysis”, 2006, to appear in Proceedings of SCAN’06. 2. M.H. van Emden and B. Moa, “Computational Euclid”, Proceedings of the 18th Canadian Conference on Computational Geometry (CCCG’06) pp. 165–168, 2006. 3. B. Moa, “Interval Constraints for Computer Graphics: Plots, Tessellations, and Ani- mations”, Proceedings of CSP Techniques with Immediate Applications (CSPIA04), 2004. 4. B. Moa, “An Interval Constraint Approach to Neural Networks”, Proceedings of The 2005 International Conference on Machine Learning, Models, Technologies and Ap- plications (MLMTA05), pp. 87–96, 2005. 5. M.H. van Emden and B. Moa, “Solving non-linear Diophantine equations using re- laxation and interval constraints”, SIAM Conference on Discrete Mathematics, 2006. 6. B. Moa, “A Surprising Approach to Graceful Labeling of Trees”, SIAM Conference on Discrete Mathematics, 2006. 7. B. Moa, “Interval Constraints for Flight-Envelope Clearance”, COOP report, Depart- ment of Computer Science, University of Victoria, 2006. 8. B. Moa, “Modeling and Simulating Wakan’s Art”, COOP report, Department of Computer Science, University of Victoria, 2006. 9. B. Moa, “Constraint Programming Tools”, COOP report, Department of Computer Science, University of Victoria, 2004. 10. M.H. van Emden and B. Moa, “Termination criteria in the Moore-Skelboe Algorithm for Global Optimization by Interval Arithmetic”, Proceedings of Frontiers in Global Optimization, 2003. 11. M.H. van Emden and B. Moa, “Propagation by Selective Initialization and Its Appli- cation to Numerical Constraint Satisfaction Problems”, Technical report, Department of Computer Science, University of Victoria, 2004. 12. M.H. van Emden and B. Moa, “Using Propagation for Solving Complex Arithmetic Constraints”, Technical report, Department of Computer Science, University of Vic- toria, 2004. 13. M.H. van Emden, B. Moa and S.C. Somosan, “Functions to Support Input and Output of Intervals”, submitted to Journal of Reliable Computing, 2007. 14. B. Moa, “Interval Constraints with Exploration for Global Optimization”, submitted to Journal of Global Optimization, 2007. 15. B. Moa, “On Box Consistency, Box Satisfaction, and Box Unsatisfaction”, submitted to Journal of Reliable Computing, 2007. University of Victoria Partial Copyright License

I hereby grant the right to lend my dissertation to users of the University of Victoria Library, and to make single copies only for such users or in response to a request from the Library of any other university, or similar institution, on its behalf or for one of its users. I further agree that permission for extensive copying of this dissertation for scholarly purposes may be granted by me or a member of the University designated by me. It is understood that copying or publication of this dissertation for financial gain shall not be allowed without my written permission.

Title of Dissertation: Interval Methods for Global Optimization.

Author: Belaid Moa June 28, 2007 DISSERTATION WITHHOLDING FORM

At our request, the commencement of the period for which the partial licence shall operate shall be delayed from June 28, 2007 for a period of at least six months.

(Supervisor)

(Department Chairman)

(Dean of Graduate Studies)

(Signature of Author)

(Date)

Date Submitted to the Dean’s Office: