Front. Electr. Electron. Eng. DOI 10.1007/s11460-012-0170-6
Alan YUILLE, Xuming HE Probabilistic models of vision and max-margin methods
c Higher Education Press and Springer-Verlag Berlin Heidelberg 2012
Abstract It is attractive to formulate problems in estimation. We speculate that better results can be ob- computer vision and related fields in term of probabilis- tained by using better bounds and approximations. tic estimation where the probability models are defined over graphs, such as grammars. The graphical struc- Keywords structured prediction, max-margin learn- tures, and the state variables defined over them, give ing, probabilistic models, loss function a rich knowledge representation which can describe the complex structures of objects and images. The proba- bility distributions defined over the graphs capture the 1 Introduction statistical variability of these structures. These proba- bility models can be learnt from training data with lim- There has recently been major progress in the devel- ited amounts of supervision. But learning these models opment of probability models defined over structured suffers from the difficulty of evaluating the normaliza- representations including graphs and grammars [1–7]. tion constant, or partition function, of the probability This interdisciplinary enterprize combines computer sci- distributions which can be extremely computationally ence expertise on representations with statistical mod- demanding. This paper shows that by placing bounds eling of probabilities. For example, natural language re- on the normalization constant we can obtain compu- searchers have defined stochastic context free grammars tationally tractable approximations. Surprisingly, for (SCFGs) by putting probability distributions over con- certain choices of loss functions, we obtain many of the text free grammars which encode the hierarchy of nouns, standard max-margin criteria used in support vector nouns phrases, and so on. The same approaches can be machines (SVMs) and hence we reduce the learning extended to more powerful grammars [2,7]. But the rep- to standard machine learning methods. We show that resentational advantages of these models are balanced many machine learning methods can be obtained in by their computational requirements and, in particular, this way as approximations to probabilistic methods whether they admit efficient learning algorithms. including multi-class max-margin, ordinal regression, In particular, it is attractive to formulate vision as max-margin Markov networks and parsers, multiple- probabilistic inference on structured probability repre- instance learning, and latent SVM. We illustrate this sentations. This seems both a natural way in which to work by computer vision applications including image deal with the complexities and ambiguities of image pat- labeling, object detection and localization, and motion terns [8,9] and also fits into a more unified framework for cognition and artificial intelligence [10]. But vision is a particularly challenging problem to formulate in this Received October 10, 2011; accepted November 22, 2011 manner. The complexity of vision requires distributions Alan YUILLE ,XumingHE defined over very complicated structures and requires Department of Statistics, University of California at Los Angeles, principles such as compositionality and the use of graph- Los Angeles, CA 90095, USA ical models with variable topology [11] and stochastic E-mail: [email protected] grammars [5]. We will give a brief review of probabilistic models of vision in Section 2. But these complex prob- Alan YUILLE ability models also require significant computational re- Department of Brain and Cognitive Engineering, Korea Univer- quirements both to perform inference and for learning. sity, Seoul 136-701, Korea By contrast, machine learning techniques are often de- Xuming HE signed to be computationally tractable [12,13] although NICTA Canberra Research Laboratory, Canberra, ACT 2601, they are not as well-principled as probabilistic methods Australia for dealing with complex tasks [14]. But how can we take 2 Front. Electr. Electron. Eng. advantage of the representational power of probabilistic Then in Section 6 we extend our approach to other ex- models and the computational tractability of machine amples where hidden/latent variables are present. Then learning algorithms? There have been attempts to show we illustrate this work on examples from our previous relations between them, briefly reviewed in Section 2. work in computer vision which originally motivated the There has been success in relating AdaBoost to proba- bounds in this paper, see Section 7. bilistic learning [15–18] but despite some interesting re- sults [14,19–22]. The attempts to show connections be- tween max-margin methods [13] and probabilities have not as yet made a big impact on the community. In this paper, our starting point is a series of works published in the last few years [23–27], which formu- lated complex vision problems, such as the detection and parsing of objects, the labeling of image pixels, and the estimation of motion, in terms of probability models based on compositional principles. These models then used approximations which enabled the models to be learnt by machine learning techniques based on max- margin criteria. In this paper we clarify the nature of the approximations and show that they can be derived rigorously as bounds on the probabilistic criteria. For certain probabilistic criteria, involving appropriate loss functions, this obtains many standard max-margin crite- ria with or without hidden variables. This yields interest- ing relationships between probabilistic and margin learn- ing concepts. For example, machine learning attempts to Fig. 1 Max-margin methods try to maximize the margin to en- maximize the margin in order to improve generalization courage generalization preferring large margins (a) to smaller mar- to novel data while probabilistic methods attempt to gins (b). For the probability distributions the corresponding intu- maximize the variance of the distribution in order to ob- ition is that we will favors models with large covariance (c) over tain a similar effect (i.e., prevents over-fitting the data), models with smaller covariance (d) in order to discourage over- fitting the data. see Fig. 1. The notion of support vectors, i.e., the results that the learnt solution depends only on a subset of the training data, arises only from the approximations made 2 Background to the partition function. We note that some results of this type are known in the natural language community [7]. 2.1 Probabilistic models Our approach has several advantages. We can relate probabilistic models to existing machine learning meth- There has been a long history in computer vision of mod- ods. We can formulate models of more complex phenom- eling objects by deformable templates defined in terms of ena using the well-principled probabilistic approach and energies [28–30]. More recent models formulate objects then obtain novel machine learning methods. We can im- and images in terms of probabilistic models defined over prove machine learning methods, and get better bounds structured representations. There is an enormous litera- for probability models, by making better approximations ture on these topics and only mention here a small sam- and bounds. ple of some of the most closely related work. The structure of this paper is as follows. We first One stream of research follows the inspiration of briefly mention some relevant probabilistic models from Grenander’s pattern theory [8,9] leading to generative the computer vision literature and then sketch work models build using compositionality [11] or stochas- which relates machine learning and probabilistic meth- tic grammars [5]. These generative approaches are ods, see Section 2. Next we describe probabilistic mod- computationally demanding despite the exploitation of els in Section 3 and introduce the types of models we discriminative models to perform inference by data will illustrate in the computer vision section. Then we driven Markov Chain Monte Carlo [31]. Another stream describe the learning criteria used in probabilistic meth- of research leads to discriminative models as exemplified ods and machine learning and the relationships between by conditional random fields [32,33]. These have moti- them, see Section 4. Next in Section 5 we introduce vated a whole series of works on hierarchical models [34– our approach of deriving bounds for probabilistic models 36]. A particularly important example are latent support by considering models without hidden/latent variables. vector machine (SVM)’s [37], which we will mention in Alan YUILLE et al. Probabilistic models of vision and max-margin methods 3
Section 5. Other classes of grammatical models includes tra temperature hyper-parameter, but did not establish stochastic grammatical models for text [38] and hierar- a bounding relations between the different criteria. chical models of objects by Ahuja and Todorovic [39,40]. In all this work, however, the loss functions are com- Yet another related strand of research are deep belief bined into the probability distributions while this pa- networks [41] and the models by Amit and his colleagues per keeps them separate which give greater flexibility [42]. for practical applications. Neither do the studies above In this paper we will concern ourselves with a class of address the types of complex models used for our com- models of objects and images [23–27], which are built puter vision applications. using the concepts of compositionality and stochastic grammars [5,11]. But these models are learnt discrim- inatively using the techniques described in this paper. 3 Probabilistic models Space did not enable us to also describe the closely re- lated work by Kokkinos and Yuille [43,44], which makes We consider probability models formulated over graphs use of multiple instance learning, see Subsection 6.2. G =(V, E)whereV denotes the vertices and E denotes edges. State variables yμ are defined at the nodes μ ∈V. The variable y = {yμ : μ ∈V}describes the state of 2.2 Machine learning and its relationship to the entire graph. The edges E in the graph specify which probabilistic modeling nodes are directly connected and define the cliques Cl, i.e., for all μ1,μ2 ∈Cl then (μ1,μ2) ∈E. We index Cl by We briefly summarize previous work which addresses the α and let yα represent the state of all variable in clique α. relationship between probabilistic modeling and related The computer vision models we consider in this paper methods developed by the machine learning community. has been built from elementary AND and OR cliques, In particular, there have been several studies showing see Fig. 2(a), which can be combined into larger models that AdaBoost is very similar to sigmoid regression. For by composition, see Fig. 2(b). For example, the head of example, Hastie et al. [15] showed that AdaBoost con- the baseball player in Fig. 3(b) can either be upright or verges to a conditional distribution in the limit of large oriented, while the object can be modeled by AND-ing amounts of data. Subsequent work [16–18] showed more the legs, the torso and the head. The specific models are detailed relations between the two methods, in particu- given in Refs. [23–27] and will be given as illustrations lar when an l1-norm regularizer is imposed on the weight in Section 7. parameters in AdaBoost. Potential functions φα(yα, x) and parameters wα are This paper is concerned with max-margin tech- defined over the cliques α ∈Cl (note that these poten- niques. The relationship between regularization and tials allow direct input from the data x). AND and OR max-margin criteria have long been known and are nodes will have different types of potential functions. We briefly discussed in Ref. [21] which also mentions how also define potentials which depend on the states of indi- regularization can be re-interpreted as Bayesian estima- vidual graph nodes and are dependent only on the input tion. Earlier related research on this topic includes the x. seminar work of Wahba [19]. Seeger [20] gives a thor- Examples of the probability models are given in Fig. 3. ough discussion of these issues and, in particular, how The nodes μ ∈Vrepresent subparts of the object where kernel methods can relate to Gaussian processors (e.g., states yμ specify whether the part is present/absent and, covariance matrix over finite sets of points can be used if it is present, its position and other attributes (e.g., to obtain either Gaussian process models or kernels for orientation and size). For our computer vision applica- SVMs). Alternative work [45] is also closely related to tions the graphs are typically organized in layers where Gaussian processes and provides probabilistic classifica- a node at one layer is connected to a subset of nodes tion using an expectation-maximization (EM) training at the lower level forming a clique. These cliques are of algorithm. Zhu and Xing [22] uses a maximum entropy two types: (i) AND-cliques where the upper node repre- discrimination framework for learning probabilistic mod- sent the position/attributes of a part which is composed els. More recently, Franc et al. [14] studied the binary- from subparts represented by the lower nodes, and (ii) classification case and showed that exact equivalence OR-cliques where the upper node takes the same state could be obtained between SVMs and probabilistic lean- as one of the lower nodes, i.e., it chooses between them. ing for exponential models if certain model parameters, We specify a conditional distribution: such as the modulus of the weight vector, were treated 1 as hyper-parameters and were estimated differently than P (y|x; w)= × exp{ wμ · φ(yμ, x) Z[w, x] the other model parameters. In Ref. [46], Pletscher et μ∈V al. also suggested a unified view of log-loss and max- + wα · φα(yα, x)}. (1) margin loss for structured model by introducing an ex- α∈Cl 4 Front. Electr. Electron. Eng.
and hence inference can be performed by the junction trees algorithm. During training/learning, we are given training ex- amples which specify the states of the input {xi : i =1, 2,...,N} and a set/subset of the state variables {yi : i =1, 2,...,N}. It is convenient to divide this into two cases. The first case is when all the {yi} are speci- fied. In the second, by a slight abuse of notation, we di- vide the states variables into {(yi, hi):i =1, 2,...,N} where the states {yi} are specified during learning and the states {hi} are not. In both cases we end up with exponential models of respective form:
exp{w · φ(x, y)} P (y|x, w)= , (2) Z[x, w] exp{w · φ(x, y, h)} P (y, h|x, w)= , (3) Z[x, w]
where w and φ(·) are vectors representing all the pa- rameters and potentials described above. In addition we have loss functions l(y, yT)andl(y, h, yT) respectively. However, learning the parameters of the models in Eqs. (2) and (3) is computationally demanding due to complex forms of the normalization terms, or partition functions, Z[x, w]. This motivates the need for approxi- Fig. 2 The basic components of the probability graphs are AND mate methods which simplify the computation in order and OR cliques (a). In an AND cliques the parent state yp rep- resents a part which is composed of elementary parts represented to perform learning. by y1,y2,y3. For OR cliques, the state of the parent node yp must select one of the child nodes y1 or y2. Probability models are con- structed by treating AND and OR cliques are elementary compo- 4 Max-margin criteria and probabilistic nents which can be combined to build arbitrarily complex models (b). learning criteria
These models also involve a loss function l(y, yT)which In this section we consider the max-margin criteria and is the cost of making decision y if the true state of the variants of probabilistic criteria used for learning. The y graph should be T. Hence inference requires estimat- input is a dataset of examples. D = {(xi, yi):i = ing yˆ =argminy z l(y, z)P (z|x : w). Depending on 1, 2,...,N}.Herex is the input, e.g., an image, and the applications we use dynamic programming, junction y is the output which can either be multi-class, e.g., trees, or approximate algorithms (e.g., belief propaga- y ∈{1, 2,...,M} for some M, or a vector y =(y1, y tion) to compute ˆ. For example, the probabilistic model y2,...,yM ), whose components are multi-valued (as shown in Fig. 2(b) contains closed loops of limited size were used for learning Markov models [47], learning
Fig. 3 AND/OR tree graphical models are capable of representing 100 different poses of a baseball player using a compact representation which exploits part sharing (a), and are efficient in representing different appearances of an object (b) [24]. Alan YUILLE et al. Probabilistic models of vision and max-margin methods 5 grammars for natural language [48], and in some our How do the two criteria given in Eqs. (5) and (6) re- computer vision where, for example, yν represents the late to learning probabilistic models? The forms of Eqs. position and attribute of the object part represented by (5) and (6) are suggestive of the exponential models de- node ν). scribed in Section 3. There are other similarities, e.g., The max-machine criteria are designed to seek a deci- the inference algorithms used for latent SVMs are anal- sion rule yˆ(xi; w)offormyˆ(x; w) = arg maxy w·φ(x, y) ogous to the EM algorithm for probability models, as which minimizes a convex bound of the empirical risk ob- described in Subsection 6.1. tained by summing a loss function l(y, yT) over the data Learning probability models can be formu- examples: lated as MAP estimation, i.e., seek wˆ = N arg max P (w) i P (yi|xi; w), where P (·|·; ·) is spec- N =1 ified by Eq. (2) and P (w) is a prior on the model R (yˆ)=C l(yˆ(xi; w), yi), (4) emp parameters w. This corresponds to minimizing the log i=1 posterior: |w|2 combined with a penalty term, e.g., (1/2) ,which N regularizes the solution by encouraging a large margin RMAP(w)=− log P (w) − log P (yi|xi; w). (7) 1/|w|. i=1 A standard bound is of form [49]: An alternative, which we employ in this paper, N involves introducing a loss function l(y, yi)totake 1 2 F (w)= |w| + C max{l(y, yi) into account the penalties that are paid for mak- ML 2 y i=1 ing an error in y during estimation. This yields the + w · φ(xi, y) − w · φ(xi, yi)}. (5) expected loss learning criteria REL which replaces − y |x w log P (i i; ) in Eq. (7) by the log of the expected This can be obtained from the empirical risk loss log y l(y, yˆ(x))P (y|x, w) which yields: (4) by applying the identities maxy{l(y, yi)+w · N φ(xi, y)} l(yˆ(xi; w), yi)+w · φ(xi, yˆ(xi; w)) and REL(w)=− log P (w)+ log l(y, yi)P (y|xi; w). w · φ(xi, yˆ(xi; w)) [49]. The criterion in Eq. (5) reduces i=1 y to the hinge loss criterion for binary classification [13] re- (8) placing y by y ∈{−1, +1}, expressing the loss function 1 To understand the motivation for this change, observe as l(y,y )=1− δy,y , and setting φ(x, y)= yφ(x), T T 2 that after learning we estimate y from input x by φ x x where ( ) is a potential in the data only. The crite- minimizing the expected loss to obtain yˆ(x : w)= 1 2 N rion reduces to |w| + max{0, 1 − yiw · xi},with 2 i=1 arg min l(y, yˆ(x))P (y|x, w). This reduces to MAP x w w · y solution, after training, given byy ˆ( , ) = arg maxy estimation of y in the special case that the loss function φ(x). y y − l( , T)=1 δy,yT . The SVM criterion can be extended to problems Observe that the probabilistic learning criteria, Eqs. h where there are hidden/latent variables (i.e., vari- (7) and (8), involve the entire probability distribution ables that are not specified by ground-truth during while the empirical risk and the max-margin criteria, learning). These problems are formulated by introduc- Eqs. (4) and (5), only involve functions yˆ(x; w)and φ x y h ing potential ( , , ) and expressing the solution as (yˆ(x; w), hˆ(x, w)). There are, however, similarities be- y x w hˆ x w w · φ x y h (ˆ( ; ), ( , )) = arg maxy,h ( , , ). The pa- tween the two formulations. w rameters are learnt by minimizing the latent SVM Firstly, note that yˆ(x; w)and(yˆ(x; w), hˆ(x, w)) are criterion [37,50]: both maximum likelihood (ML) estimates of y and y y|x w N from the respective probability models P ( ; )and 1 2 y h|x w L(w)= |w| + {max{l(y, h, yi)+w · φ(xi, y, hi)} P ( , ; ) given in Eqs. (1) and (3). In the limit as 2 y,h i=1 the magnitude |w| of the parameters gets large then − max w · φ(xi, yi, hi)}. (6) the probability distributions become strongly peaked h about their ML values — arg maxy w · φ(x, y)and This criterion is non-convex since the last term arg maxy,h w · φ(x, y, h). So in this limit there is little N i maxh w·φ(xi, yi, hi) is concave, while the remain- difference between using the full distributions and the =1 ˆ ing terms are convex. We note that the latent SVM used decision rules yˆ(x; w)and(yˆ(x; w), h(x, w)). in Ref. [37] is equivalent to the MI-SVM formulation Secondly, we can make a relationship be- tween the criteria using Jensen’s inequality. of Multiple Instance Learning with SVM [51]. A similar N This yields log l(y, yi)P (y|xi; w) learning criteria was also proposed by Xu et al. [52] for i=1 y N { y y } y|x w unsupervised learning problems which maximizes over i=1 y log l( , i) P ( i; ) which will become missing outputs in SVM. equal to the empirical Bayes risk in the large |w| limit 6 Front. Electr. Electron. Eng.
y y M { } but with a different loss function (i.e., replacing l( , T) bound for this expression by nb a=1 exp na when by log l(y, yT)). nb is any element of the set {na : a =1, 2,...,M}. Hence we can find some relationships between these Using these bounding methods, we can now obtain max-margin and probabilistic criteria in the large |w| two bounds for the log-risk of form: limit. But recall that |w| is the inverse of the margin and so large |w| corresponds to the small margin limit LogREL(w) ∼ E(w) which is bad for generalization and which the regularizer N + max{l(y, yi)+w · φ(xi, y)} is designed to avoid. From the probabilistic perspective, y however, large |w| corresponds to small covariance (or i=1 N low temperature in statistical physics). This is the limit − max w · φ(xi, y), (11) where the probability distribution is most peaked about y i its maximum value which we will exploit in the next sec- =1 LogR (w) E(w) tion. As we will see, there will be a tension between the EL N requirements to make |w| large to encourage good gen- + max{l(y, yi)+w · φ(xi, y)} eralization and the need to make |w| small to enable the y i=1 approximations/bounds to be tight. N − w · φ(xi, yi)+K. (12) i=1 5 Exponential models, bounds and max- margin criteria The first bound, Eq. (11), is obtained by using the first bounding method for both the log Z and log y terms. In this section we describe our basic result (which will It gives upper and lower bounds for LogR(w), indicated ∼ be extended in the following section). We consider ex- by . The second bound, Eq. (12), uses the first bound- ponential models P (y|x, w) for generating y in terms of ing method for the log y term and the second bounding input x, with parameters w,andapriorP (w): method for log Z (K is a constant). We see that the second bounds, Eq. (12), are iden- exp{w · φ(x, y)} P (y|x, w)= , tical to the structure SVM criterion provided we set Z[x, w] E(w)=|w|2 (i.e., assume a Gaussian prior). The first exp{−E(w)} method, Eq. (11), provides a tighter upper bound, as P (w)= , (9) Z well as a lower bound. But it has the disadvantage that the bound is not convex, since the final term is concave, where Z[x, w]= exp{w · φ(x, y)}. y and so it is harder to minimize. We use the EL R (w) which we re-express by setting EL The tightness of these bounds depends on the size of l(y, y )=expl(y, y ): T T |w|.Inthelarge|w| limit the upper bound of bounding method (I) becomes exact because the largest term in LogREL(w)=− log P (w) N the summation dominates. But this is the limit where + log exp{l(y, yi)}P (y|xi, w) the margin is very small and so generalization is poor. i=1 y These results enable us to relate probabilistic mod- N els to existing machine learning methods, which enables = − log P (w)+ {− log Z[xi, w] us to exploit the efficient learning algorithms such as i=1 structured perceptron [53] and structure max-margin al- +log exp{l(y, yi)+w · φ(xi, y)}}. gorithms [47–49]. Observing from this perspective, the y regularizer term which attempts to make a large mar- (10) gin corresponds to a prior on the parameter w which tries to make the covariance as large as possible, hence We relate this to the structure SVM criterion yielding good generalization. Observe that support vec- by bounding the terms of the right-hand side of tor arise in this approximation although they do not oc- x w Eq. (10). This requires bounding log Z[ i, ]and cur in the original probability model — instead they log y exp{l(y, yi)+w · φ(xi, y)}.Inbothcases, result from replacing the partition functions by its dom- this is equivalent to bounding expressions of form inant term which concentrates attention on those dat- M { } { log a=1 exp na for a set of numbers na : a = apoints with high probability which hence are near the 1, 2,...,M}. We use two methods to bound expressions decision boundary. It is also possible to follow the logic of this type: (i) We can obtain upper and lower bounds of this approach and obtain better bounds by making M { } of form nmax log a=1 exp na Mnmax,where better approximation to the partition function perhaps nmax =maxa na. (ii) We can obtain a weaker lower- by using structured variational methods [54]. Alan YUILLE et al. Probabilistic models of vision and max-margin methods 7
(16), uses the first bounding method for the log y term, 6 Further examples and bounds the log Z by using the first bounding method for h and the second bounding method for y (i.e., replace This section describes how the bounding methods can it by yi). be applied to obtain other max-margin criteria. This in- We see that the second bounds, Eq. (16), are identi- cludes methods with hidden/latent variables, such as la- cal to the latent structure SVM criterion provided we 2 tent SVM and multiple instance learning (MIL), and or- set E(w)=|w| . The first method, Eq. (15), provides dinal regression. If hidden variables are present then we a tighter upper bound, as well as a lower bound. Both bound the summations over them in a similar manner. bounds are non-convex, so the first bound may be pre- ferred because it is tighter. 6.1 Hidden variables and latent SVM There are also parallels between the algorithms used to deal with hidden variables in probability models and those used by latent SVMs. For example, a standard We now extend our analysis to deal with models with algorithm for learning a latent SVM [50] proceeds by hidden/latent variables h, which are un-specified in the decomposing Eq. (16) into concave (the last term) and training data. We extend the probabilistic model to convex parts (everything else) and using a CCCP algo- exp{w · φ(x, y, h)} rithm [55]. This yields an iterative two-step algorithm P (y, h|x, w)= , Z[x, w] which estimates the parameters w keeping the hidden h h Z[x, w]= exp{w · φ(x, y, h)}. (13) variables fixed, corresponding to treating the as ob- y,h served variables and minimizing Eq. (5), and then esti- mating the h variables by maximum a posteriori treat- We define an exponential loss function l(y, h, yT)= ing the w are known. This iterative two-step algorithm exp{l(y, h, yT)} and modify the log-risk, Eq. (10), to be is reminiscent of the EM algorithm [56] which instead would update the parameters w assuming a known dis- w LogREL( ) tribution q(h) for the hidden variables and then update = − log P (w) q(h)withw fixed. Hence, the main difference is that N EM updates a probability distribution q(h) while latent + log exp l(y, h, yi)P (y, h|xi, w) SVM updates the variable h directly. Note that would i=1 y,h be exact in the large |w| limit. = − log P (w) N 6.2 Multiple instance learning (MIL) + log exp{l(y, h, yi)+w · φ(xi, y, h)} i=1 y,h In this section, we first show that the MI-SVM formula- − log Z[x, w]. (14) tion of MIL can also be viewed as minimizing an upper Using our bounding methods, we can obtain two bound of log-risk of a corresponding probabilistic model. bounds for the log-risk: This connection allows us to extend our framework to other types of MIL objective functions such as MI-SVM LogREL(w) ∼ E(w) [51]. N We can express the MI-SVM criterion as follows [51]: + max{l(y, h, yi)+w · φ(xi, y, h)} y,h i=1 1 2 a |w| + C max{0, 1 − yi max w · xi }, (17) N a∈i 2 i∈I − max w · φ(xi, y, h), (15) y,h i=1 where I is the set of ‘bags’, i ∈Iis the bag index, and LogREL(w) E(w) a ∈ i is an instance in the bag. If a bag is ‘positive’ then N it has a single ‘active’ element, but it has no ‘active’ + max{l(y, h, yi)+w · φ(xi, y, h)} y,h elements if it is ‘negative’. i=1 We derive the MI-SVM criterion in Eq. (17) by formu- N lating it in terms of a probabilistic model with hidden − max w · φ(xi, yi, h). (16) h variables and then adapting the arguments used for la- i =1 tent SVM in the last section. First, we introduce hidden a a The first bound, Eq. (15), is obtained by using the first variables {hi } associated to the instances xi of all bags, y h a a bounding method for both variables , in the log Z such that hi = 1 if the instance is ‘active’ and hi =0 and log y,h terms. It gives upper and lower bounds if it is not ‘active’. We have constraints that either: (i) a a for LogREL(w), indicated by ∼. The second bound, Eq. a∈i hi =1ifthebagi is positive, and (ii) a∈i hi =0 8 Front. Electr. Electron. Eng. if the bag is negative. nal regression problems, or rank-SVMs (e.g., [57]). Let { x N } Next we define a joint probability distribution over the ( i,yi)i=1 be a training set, and yi denotes the rank output y and hidden variables hi: of data instance xi and takes value from {1, 2,...,R}. We want to learn a prediction function h(x)=wTx, 1 y 1 a P (y,hi|xi)= δ( + , hi ) such that yi >yj ⇐⇒ h(xi) >h(xj). Let P = {(i, j): i w xi Z ( , ) 2 2 a∈i yi >yj},andm = |P|. The original rank-SVM training a a · exp{ hi w · xi }, (18) is formulated as the following optimization problem: a∈i 1 C a ||w||2 where Zi(w, xi)=1+ exp{w · x }. We first check min + ξij , a∈i i w,ξij 0 2 m that the MAP estimate from the probability model, (i,j)∈P a T T Eq. (18), is equivalent toy ˆ(xi, w)=sgnmaxa∈i w · xi s.t. ∀(i, j) ∈P: w xi w xj +1− ξij . (20) which is the MI-SVM decision-rule. Denoting (ˆy,hˆi)= − arg max log P (y,hi|xi), we see that the equivalence holds Let zij =1ifyi >yj and zij = 1ifyi
D global optimum. For other applications, approximate in- Each node μ ∈Vhas a data potential φ (yμ, I)which ference algorithms such as belief propagation or graph links its state yμ to the input data I. These potentials cuts can be used. The learning can be done by making are based on filterbanks of image features. There are the approximations described in the previous sections. also clique potential terms wα · φα(yα, I)whichimpose For applications without hidden variables we can use the compatibility between state variables at different levels structure SVM criterion described in Section 5, while for of the hierarchy and priors on the frequency of different applications with hidden units we use the latent SVM segmentation-templates, the labels, and the spatial re- techniques described in Subsection 6.1. lations between labels (e.g., whether a cow is likely to Our first example is image labeling,wherethetask be adjacent to a car or to the sky). These potentials are is to assign a label s ∈Sto every pixel in the image. For specified in advance and the learning task is to estimate example, the MSRC dataset [58] has 21 different labels their parameters w, hence to select which potentials to in S. We consider a hierarchical model with a quadtree use and assign weights to them. structure in Refs. [23,25], illustrated in Fig. 4(a). The The inference can be performed by pruned dynamic state variable yμ of a node μ ∈V gives a description of programming exploiting the tree-like structure of the the image region underneath it in terms of a partition graph. The state space of the variables is big so prun- into different subregions with a label assigned to each ing is performed to remove states with low energy. The subregion. Formally, the state variables are written as learning is performed using the training dataset D = μ μ yμ =(cμ, sμ), where cμ ∈Cis a segmentation-template, {(y , x ):μ =1, 2,...,N}. The groundtruth is only which partitions the image region into subregions, and specified at the pixels, but a simple automatic meth- sμ assigns labels to each subregion. The state variables ods can be used to estimate the groundtruth for the at higher-levels of the graph give crude ‘executive sum- state variables of the graph. In this work, the loss func- maries’ descriptions of image regions which are refined tion penalizes all errors equally. The probabilistic crite- at lower-levels, see Fig. 4(c). rion for learning the parameters w and its upper bound can be expressed in the form described in Section 5. For this application, the authors choose to solve the problem by minimizing the upper bound in the primal space us- ing the structure perceptron algorithm [53]. The perfor- mance of this method was among the start-of-the-art on the MSRC dataset. The labeling performance was 74.4% for class-average accuracy and 81.4 % for pixel-average accuracy, using the same set up and criteria reported in Ref. [58], which were significant improvements (by 7.3% and 5.5%) over alternative methods at time of submis- sion. Object detection and pose estimation.An- other example is to detect articulated objects which have multiple poses, such as people and horses, see Fig. 3 [24]. The object is represented by an AND/OR graph [5]. Essentially the object is constructed by composition which involves AND-ing and OR-ing elementary parts together to form the object. The OR-ing operation al- lows the model to deal with different poses of the ob- ject. This is formulated as probability defined on a graph with state variables yμ =(pμ,tμ), where pμ represents the pose (i.e., position, orientation, and scale) of an ob- ject part, and tμ is a binary variable which indicates if the part is selected or not. The layers of the graph alternate between AND nodes, for which the part is a composition of the sub-parts of its child nodes (see Fig.
Fig. 4 (a) The graph structure of the recursive compositional 3), and OR nodes which requires a node to select one model used for image labeling in Ref. [23]. The state variable of its child nodes (i.e., a ‘head’ must be ‘head-up’ or at each graph node describes an image region by a labeled ‘head-down’). The t variables perform the selection for segmentation-template. (b) Thirty segmentation-templates. (c) the OR nodes, similar to switch variables [5]. The graph The states at higher-levels of the graph give crude descriptions of image regions which are refined at lower-levels. (d) Examples structure has different topology due to the state of the on the MSRC dataset (image, groundtruth, model output). t variables (i.e., the selections made at the OR nodes). 10 Front. Electr. Electron. Eng.
This means that a graph containing only 40 nodes can The inference requires estimating the best state of the have 100 different topologies, which correspond to 100 six possible hierarchies which can be done by dynamic different poses. An alternative strategy would be to have programming over each hierarchy, followed by exhaus- 100 different models — one for each pose of the object — tive search over the six possibilities. The learning in- and perform pose estimation by selection between these volves hidden variables since the training data is of form different models. The AND/OR graph is more compact D = {(Ii, yi):i =1, 2,...,N}, and only says whether and efficient, because it is able to share parts (i.e., the an object is present or not in the image (i.e., it does not most elementary components) between different poses. specify the mixture component V or the positions {pμ} The potentials φα(yα) impose spatial relations on the of the object parts). With a ‘0-1’ loss function, the learn- relative positions of parts, their composition from more ing criterion can be expressed in the form given in Sub- elementary parts, and the selection choices made at OR section 6.1. Again, we can view the learning objective as D nodes. The data potentials φ (yμ, I) relate the node at an upper bound, which was optimized by a variant of Yu level l = 1 to the image (e.g., by encouraging the bound- and Joachims’s procedure [50]. The performance of this aries of parts to correspond to edge-type features in the approach achieves the state-of-the-art and has recently image). obtained second prize in the PASCAL object detection The graph structure for this model contains some challenge in 2010. closed loops because of the graph edges connecting sib- The third task is motion estimation,whichaims lings (e.g., the spatial representations between parts at at estimating the motion flow between two images. The the same scale). However, the number of closed loops is model used as our example [27] is defined on a hierarchi- small enough that we can perform inference using dy- cal graph with closed loops (see Fig. 5(c)). The nodes of namic programming with the junction trees algorithm. the graph represent the motion at different scales, so it Thesizeofthestatespace,however,isverylargebe- defines motion priors and contextual effects at a range of cause there are many possible values for the pose pμ of scales. The potential functions of this model contain data each part. Therefore, we perform pruning based on the terms which require the corresponding points in the two energy and by surround suppression (penalizing states images have similar intensity properties and a hierar- which are too similar to each other). We emphasize that chical prior which encourages smoothness and slowness we are performing inference over the state variables in- of the velocity field. The overall distribution of motion cluding the topology (i.e., the tμ’s), which means that given image pair has the same form as Eq. (1). we do inference over 100 different models efficiently be The inference is done by a variant of loops belief prop- exploiting the re-use of the elementary parts. agation with adaptive quantization of velocity space. For The loss function l(y, yT) penalizes configurations learning algorithm, the same loss function as in image where the estimated pose pμ of a node μ ∈Vdiffers labeling is employed (after quantization), and the up- from an image Ii differs from the correct (ground-truth) per bound is minimized by structure perceptron based pose pmu,i by more than a threshold. The groundtruth on the groundtruth from the Middlesbury dataset [59]. for the baseball database is specified only for the nodes This model is shown to be able to handle both short- at level l = 1, but we can estimate it for the higher range and long-range motion, and adequately account level nodes also. Learning can be approximated by the for human performance in psychophysical experiments SVM criterion without hidden variables, as described in [27]. Section 5. For this application, we choose to perform learning in the dual space using structured max-margin algorithms [47–49]. 8 Summary Our next example is an extension of the latent SVM model successfully applied to object detection and The purpose of this paper is to draw relationships be- localization [26]. Each object model is a mixture of tween probabilistic methods and machine learning ap- six different hierarchical graphs, shown in Fig. 5(a). The proaches which use max-margin criteria. We show that nodes have variables pμ which indicate the positions of several standard max-margin criteria, with and without the parts, and there is a variable V which specifies which hidden/latent variables, can be obtained as upper or of the six graphs is selected (corresponding to different lower bounds to the probabilistic criteria. This perspec- views/poses). There is also a variable y which specifies tive suggests that machine learning approaches can be if the object is detected. In the object model, there are applied to problems that can be formulated as proba- potentials wα · φα(yα) which impose spatial relations bilistic inference over structured representations (e.g., on the relative positions of the object parts. There are graphs and grammars), and hence can be applied to D also data potentials φ (yμ, I), which relate the position a range of computer vision problems as approxima- of each part to the image I, based on HOG and SIFT tions. These relationships also suggest alternative ma- features. chine learning criteria which may be more effective in Alan YUILLE et al. Probabilistic models of vision and max-margin methods 11
Fig. 5 (a) A hierarchical model of parts, which are used as mixture components to model different poses of objects [26]. (b) These models can detect objects from different viewpoints, with changes in geometry (spatial warps) and even partial occlusion. They are trained by latent SVM techniques where the groundtruth only specifies that an object is present or not. (c) A hierarchical model of motion in one dimension [27]. Each node represents motion at different locations and scales. A child node can have multiple parents, and the prior constraints on motion are expressed by parent-child interactions. some cases than the standard ones (e.g., by using tighter Acknowledgements We would like to acknowledge support upper bounds). for this project from the National Science Foundation (NSF grant IIS-9988642), the Multidisciplinary Research Program of the De- What are the trade-offs, from a probabilistic perspec- partment of Defense (MURI N00014-00-1-0637), and by the Ko- tive, to using these types of bounds? There are clearly rean Ministry of Education, Science, and Technology, under the cases where making these bounds could cause serious National Research Foundation WCU program R31-10008. problems, e.g., replacing the sum over hidden variables in the EM algorithm by their maximum value. However, on the other hand, it is possible that the bounds (i.e., References the max-margin criterion) may be more robust than the 1. Pearl J. Probabilistic Reasoning in Intelligent Systems. probabilistic methods. The argument, original due to Morgan-Kaufmann, 1988 Vapnik [12], is that learning algorithms should pay most 2. Manning C D, Sch¨utze H. Foundations of Statistical Natural attention to regions near the decision boundaries rather Language Processing. Cambridge, MA: MIT Press, 1999 than wasting resources trying to fit the parameters of 3. Heckerman D. A tutorial on learning with Bayesian networks. the models away from the boundaries. In particular, if Learning in Graphical Model. 1999, 301–354 our probability model for the data is wrong, or non- 4. Russell S, Norvig P. Artificial Intelligence: A Modern Ap- robust, then attempting to fit the data perfectly by the proach. 2nd ed. Prentice Hall, 2003 probability models may cause errors as well as be com- 5. Zhu S C, Mumford D. A stochastic grammar of images. Foun- putationally demanding. We note, however, that these dations and Trends in Computer Graphics and Vision, 2006, bounds become exact in the large |w| limit, where the 2(4): 259–362 margin tends to zero, which is the situations where poor 6. Tenenbaum J B, Griffiths T L, Kemp C. Theory-based generalization is expected. Bayesian models of inductive learning and reasoning. Trends 12 Front. Electr. Electron. Eng.
in Cognitive Sciences, 2006, 10(7): 309–318 the 2010 IEEE Conference on Computer Vision and Pattern 7. Smith N A. Linguistic Structure Prediction. Synthesis Lec- Recognition. 2010 tures on Human Language Technologies. Morgan and Clay- 27. Wu S, He X, Lu H, Yuille A L. A unified model of short- pool, 2011 range and long-range motion perception. In: Proceedings of 8. Grenander U. Pattern Synthesis: Lectures in Pattern Theory Neural Information Processing Systems (NIPS 2010). 2010 1. New York, NY: Springer, 1976 28. Fischler M A, Elschlager R A. The representation and match- 9. Grenander U. Pattern Analysis: Lectures in Pattern Theory ing of pictorial structures. IEEE Transactions on Computers, 2. New York, NY: Springer, 1978 1973, 22(1): 67–92 29. Yuille A L, Hallinan P W, Cohen D S. Feature extraction 10. Tenenbaum J B, Yuille A L. IPAM Summer School: The from faces using deformable templates. International Jour- Mathematics of the Mind. IPAM, UCLA, 2007 nal of Computer Vision, 1992, 8(2): 99–111 11. Jin Y, Geman S. Context and hierarchy in a probabilistic im- 30. Cootes T F, Edwards G J, Taylor C J. Active appearance age model. In: Proceedings of the 2006 IEEE Conference on models. Lecture Notes in Computer Science, 1998, 1407: Computer Vision and Pattern Recognition. 2006, 2145–2152 484–498 12. Vapnik V N. The Nature of Statistical Learning Theory. New 31. Tu Z, Chen X, Yuille A L, Zhu S C. Image parsing: Unifying York, NY: Springer-Verlag, 1995 segmentation, detection, and recognition. In: Proceedings of 13. Sch¨olkopf B, Smola A. Leaning with Kernels. MIT Press, the 9th IEEE International Conference on Computer Vision. 2001 2003, 1: 18–25 14. Franc V, Zien A, Sch¨olkopf B. Support vector machines as 32. Lafferty J D, McCallum A, Pereira F C N. Conditional ran- probabilistic models. In: Proceedings of the 28th Interna- dom fields: Probabilistic models for segmenting and label- tional Conference on Machine Learning (ICML-11). 2011, ing sequence data. In: Proceedings of the 18th International 665–672 Conference on Machine Learning. 2001, 282–289 15. Hastie T, Tibshirani R, Feldman J. The Elements of Statis- 33. Kumar S, Hebert M. A hierarchical field framework for uni- tical Learning. 1st ed. Springer, 2001 fied context-based classification. In: Proceedings of the 10th 16. Lebanon G, Lafferty J. Boosting and maximum likelihood for IEEE International Conference on Computer Vision. 2005, exponential models. In: Proceedings of Neural Information 2: 1284–1291 Processing Systems (NIPS 2001), 2001 34. He X, Zemel R S, Carreira-Perpi˜n´an M A.´ Multiscale con- 17. R¨atsch G, Onoda T, M¨uller K R. Soft margins for AdaBoost. ditional random fields for image labeling. In: Proceedings of Machine Learning, 2001, 42(3): 287–320 the 2004 IEEE Conference on Computer Vision and Pattern Recognition. 2004, 2: 695–702 18. R¨atsch G, Mika S, Sch¨olkopf B, M¨uller K R. Constructing boosting algorithms from SVMs: An application to one-class 35. Winn J M, Jojic N. LOCUS: Learning object classes with un- classification. IEEE Transactions on Pattern Analysis and supervised segmentation. In: Proceedings of the 10th IEEE Machine Intelligence, 2002, 24(9): 1184–1199 International Conference on Computer Vision. 2005, 1: 756– 763 19. Wahba G. Spline Models for Observational Data. SIAM CBMS-NSF Regional Conference Series in Applied Mathe- 36. Sudderth E B, Torralba A B, Freeman W T, Willsky A S. matics. Volume 59. SIAM, Philadelphia, PA, 1990 Learning hierarchical models of scenes, objects, and parts. In: Proceedings of the 10th IEEE International Conference 20. Seeger M. Bayesian model selection for support vector on Computer Vision. 2005, 2: 1331–1338 machines, Gaussian processes and other kernel classifiers. 37. Felzenszwalb P, Mcallester D, Ramanan D. A discrimina- In: Proceedings of Neural Information Processing Systems tively trained, multiscale, deformable part model. In: Pro- (NIPS 1999). 1999, 603–609 ceedings of the 2008 IEEE Conference on Computer Vision 21. Poggio T, Smale S. The mathematics of learning: Dealing and Pattern Recognition. 2008 with data. American Mathematical Society, 2003 38. Shilman M, Liang P, Viola P A. Learning non-generative 22. Zhu J, Xing E P. Maximum entropy discrimination Markov grammatical models for document analysis. In: Proceedings networks. Journal of Machine Learning Research, 2009, 10: of the 10th IEEE International Conference on Computer Vi- 2531–2569 sion. 2005, 2: 962–969 23. Zhu L, Chen Y, Lin Y, Yuille A L. A hierarchical image 39. Ahuja N, Todorovic S. Learning the taxonomy and models of model for polynomial-time 2D parsing. In: Proceedings of categories present in arbitrary image. In: Proceedings of the Neural Information Processing Systems (NIPS 2008). 2008 11th IEEE International Conference on Computer Vision. 24. Zhu L, Chen Y, Lu Y, Lin C, Yuille A L. Max margin 2007 AND/OR graph learning for parsing the human body. In: 40. Todorovic S, Ahuja N. Region-based hierarchical image Proceedings of IEEE Conference on Computer Vision and matching. International Journal of Computer Vision, 2008, Pattern Recognition. 2008 78(1): 47–66 25. Zhu L, Chen Y, Yuille A L. Learning a hierarchical de- 41. Hinton G E, Osindero S, Teh Y W. A fast learning algo- formable template for rapid deformable object parsing. IEEE rithm for deep belief nets. Neural Computation, 2006, 18(7): Transactions on Pattern Analysis and Machine Intelligence, 1527–1554 2010, 32(6): 1029–1043 42. Amit Y, Geman D, Fan X. A coarse-to-fine strategy for mul- 26. Zhu L, Chen Y, Yuille A L, Freeman W. Latent hierarchical ticlass shape detection. IEEE Transactions on Pattern Anal- structure learning for object detection. In: Proceedings of ysis and Machine Intelligence, 2004, 26(12): 1606–1621 Alan YUILLE et al. Probabilistic models of vision and max-margin methods 13
43. Kokkinos I, Yuille A L. Hop: Hierarchical object parsing. Alan YUILLE received his B.A. In: Proceedings of the 2009 IEEE Conference on Computer degree in mathematics from the Vision and Pattern Recognition. 2009, 802–809 University of Cambridge in 1976, 44. Kokkinos I, Yuille A L. Inference and learning with hierarchi- cal shape models. International Journal of Computer Vision, and completed his Ph.D. de- 2011, 93(2): 201–225 gree in theoretical physics at 45. Tipping M E. Sparse Bayesian learning and the relevance Cambridge in 1980 studying un- vector machine. Journal of Machine Learning Research, 2001, 1: 211–244 der Stephen Hawking. Following 46. Pletscher P, Ong C S, Buhmann J M. Entropy and margin this, he held a postdoc position maximization for structured output learning. In: Proceed- with the Physics Department, ings of the 2010 European Conference on Machine Learning and Knowledge Discovery in Databases. 2010 University of Texas at Austin, and the Institute for The- 47. Taskar B, Guestrin C, Koller D. Max-margin Markov net- oretical Physics, Santa Barbara. He then joined the Ar- works. In: Proceedings of Neural Information Processing tificial Intelligence Laboratory at MIT (1982–1986), and Systems (NIPS 2003). 2003 followed this with a faculty position in the Division of 48. Taskar B, Klein D, Collins M, Koller D, Manning C. Max- margin parsing. In: Proceedings of the 2004 Conference on Applied Sciences at Harvard (1986–1995), rising to the Empirical Methods in Natural Language Processing. 2004 position of associate professor. From 1995–2002 Alan 49. Tsochantaridis I, Joachims T, Hofmann T, Altun Y. Large worked as a senior scientist at the Smith-Kettlewell Eye margin methods for structured and interdependent output variables. Journal of Machine Learning Research, 2006, 6: Research Institute in San Francisco. In 2002 he accepted 1453–1484 a position as full professor in the Department of Statis- 50. Yu C N, Joachims T. Learning structural SVMs with latent tics at the University of California at Los Angeles. He variables. In: Proceedings of the 26th International Confer- has over one hundred and fifty peer-reviewed publica- ence on Machine Learning. 2009 51. Andrews S, Tsochantaridis I, Hofmann T. Support vector tions in vision, neural networks, and physics, and has machines for multiple-instance learning. In: Proceedings of co-authored two books: Data Fusion for Sensory Infor- Neural Information Processing Systems (NIPS 2002). 2002 mation Processing Systems (withJ.J.Clark)andTwo- 52. Xu L, Wilkinson D, Schuurmans D. Discriminative unsuper- vised learning of structured predictors. In: Proceedings of the and Three-Dimensional Patterns of the Face (with P. W. 23rd International Conference on Machine Learning. 2006 Hallinan, G. G. Gordon, P. J. Giblin and D. B. Mum- 53. Collins M. Discriminative training methods for hidden ford); he also co-edited the book Active Vision (with Markov models: Theory and experiments with perceptron A. Blake). He has won several academic prizes and is a algorithms. In: Proceedings of the ACL02 Conference on Empirical Methods in Natural Language Processing. 2002 Fellow of IEEE. 54. Yedidia J S, Freeman W T, Weiss Y. Constructing free- energy approximations and generalized belief propagation al- Xuming HE received the B.Sc. gorithms. IEEE Transactions on Information Theory, 2005, 51(7): 2282–2312 and M.Sc. degrees in electron- 55. Yuille A L, Rangarajan A. The concave-convex procedure ics engineering from Shanghai (CCCP). In: Proceedings of Neural Information Processing Jiao Tong University in 1998 and Systems (NIPS 2001). 2001 2001, and Ph.D. degree in com- 56. Dempster A, Laird N, Rubin D. Maximum likelihood from incomplete data via the EM algorithm. Journal of the Royal puter science from the Univer- Statistical Society, Series B: Methodological, 1977, 39(1): 1– sity of Toronto in 2008. He held 38 postdoctoral position at the Uni- 57. Joachims T. Training linear SVMs in linear time. In: Pro- ceedings of the 12th ACM SIGKDD International Conference versity of California at Los Angeles (UCLA) before join- on Knowledge Discovery and Data Mining. 2006 ing National ICT Australia (NICTA). He is currently a 58. Shotton J, Winn J, Rother C, Criminisi A. TextonBoost: researcher in the Canberra Lab of NICTA and also an Joint appearance, shape and context modeling for multi-class object recognition and segmentation. In: Proceedings of Eu- adjunct Research Fellow of Engineering Department at ropean Conference on Computer Vision. 2006 the Australian National University (ANU). His research 59. BakerS,RothS,ScharsteinD,BlackMJ,LewisJP,Szeliski interests include image segmentation and labeling, visual R. A database and evaluation methodology for optical flow. In: Proceedings of the 11th IEEE International Conference motion analysis, vision-based navigation, and undirected on Computer Vision. 2007 graphical models.