Submodularity Cuts and Applications

Submodularity Cuts and Applications

Submodularity Cuts and Applications Yoshinobu Kawahara¤ Kiyohito Nagano The Inst. of Scientific and Industrial Res. (ISIR), Dept. of Math. and Comp. Sci., Osaka Univ., Japan Tokyo Inst. of Technology, Japan [email protected] [email protected] Koji Tsuda Jeff A. Bilmes Comp. Bio. Research Center, Dept. of Electrical Engineering, AIST, Japan Univ. of Washington, USA [email protected] [email protected] Abstract Several key problems in machine learning, such as feature selection and active learning, can be formulated as submodular set function maximization. We present herein a novel algorithm for maximizing a submodular set function under a car- dinality constraint — the algorithm is based on a cutting-plane method and is implemented as an iterative small-scale binary-integer linear programming proce- dure. It is well known that this problem is NP-hard, and the approximation factor achieved by the greedy algorithm is the theoretical limit for polynomial time. As for (non-polynomial time) exact algorithms that perform reasonably in practice, there has been very little in the literature although the problem is quite impor- tant for many applications. Our algorithm is guaranteed to find the exact solution finitely many iterations, and it converges fast in practice due to the efficiency of the cutting-plane mechanism. Moreover, we also provide a method that produces successively decreasing upper-bounds of the optimal solution, while our algorithm provides successively increasing lower-bounds. Thus, the accuracy of the current solution can be estimated at any point, and the algorithm can be stopped early once a desired degree of tolerance is met. We evaluate our algorithm on sensor placement and feature selection applications showing good performance. 1 Introduction In many fundamental problems in machine learning, such as feature selection and active learning, we try to select a subset of a finite set so that some utility of the subset is maximized. A number of such utility functions are known to be submodular, i.e., the set function f satisfies f(S) + f(T ) ¸ f(S \ T ) + f(S [ T ) for all S; T ⊆ V , where V is a finite set [2, 5]. This type of function can be regarded as a discrete counterpart of convex functions, and includes entropy, symmetric mutual information, information gain, graph cut functions, and so on. In recent years, treating machine learning problems as submodular set function maximization (usually under some constraint, such as limited cardinality) has been addressed in the community [10, 13, 22]. In this paper, we address submodular function maximization under a cardinality constraint: max f(S) s.t. jSj · k; (1) S⊆V where V = f1; 2; : : : ; ng and k is a positive integer with k · n. Note that this formulation is considerably general and covers a broad range of problems. The main difficulty of this problem comes from a potentially exponentially large number of locally optimal solutions. In the field of ¤URL: http://www.ar.sanken.osaka-u.ac.jp/ kawahara/ 1 combinatorial optimization, it is well-known that submodular maximization is NP-hard and the approximation factor of (1 ¡ 1=e)(¼ 0:63) achieved by the greedy algorithm [19] is the theoretical limit of a polynomial-time algorithm for positive and nondecreasing submodular functions [3]. That is, in the worst case, any polynomial-time algorithm cannot give a solution whose function value is more than (1 ¡ 1=e) times larger than the optimal value unless P=NP. In recent years, it has been reported that greedy-based algorithms work well in several machine-learning problems [10, 1, 13, 22]. However, in some applications of machine learning, one seeks a solution closer to the optimum than what is guaranteed by this bound. In feature selection or sensor placement, for example, one may be willing to spend much more time in the selecting phase, since once selected, items are used many times or for a long duration. Unfortunately, there has been very little in the literature on finding exact but still practical solutions to submodular maximization [17, 14, 8]. To the best of our knowledge, the algorithm by Nemhauser and Wolsey [17] is the only way for exactly maximizing a general form of nondecreasing submodular functions (other than naive brute force). However, as stated below, this approach is inefficient even for moderate problem sizes. In this paper, we present a novel algorithm for maximizing a submodular set function under a cardi- nality constraint based on a cutting-plane method, which is implemented as an iterative small-scale binary-integer linear programming (BILP) procedure. To this end, we derive the submodularity cut, a cutting plane that cuts off the feasible sets on which the objective function values are guaranteed to be not better than current best one, and this is based on the submodularity of a function and its Lovasz´ extension [15, 16]. This cut assures convergence to the optimum in finite iterations and allows the searching for better subsets in an efficient manner so that the algorithm can be applied to suitably-sized problems. The existing algorithm [17] is infeasible for such problems since, as originally presented, it has no criterion for improving the solution efficiently at each iteration (we compare these algorithms empirically in Sect. 5.1). Moreover, we present a new way to evaluate an upper bound of the optimal value with the help of the idea of Nemhauser and Wolsey [17]. This enables us to judge the accuracy of the current best solution and to calculate an ²-optimal solution for a predetermined ² > 0 (cf. Sect. 4). In our algorithm, one needs to iteratively solve small- scale BILP (and mixed integer programming (MIP) for the upper-bound) problems, which are also NP-hard. However, due to their small size, these can be solved using efficient modern software packages such as CPLEX. Note that BILP is a special case of MIP and more efficient to solve in general, and the presented algorithm can be applied to any submodular functions while the existing one needs the nondecreasing property.1 We evaluate the proposed algorithm on the applications of sensor placement and feature selection in text classification. The remainder of the paper is organized as follows: In Sect. 2, we present submodularity cuts and give a general description of the algorithm using this cutting plane. Then, we describe a specific procedure for performing the submodularity cut algorithm in Sect. 3 and the way of updating an upper bound for calculating an ²-optimal solution in Sect. 4. And finally, we give several empirical examples in Sect. 5, and conclude the paper in Sect. 6. 2 Submodularity Cuts and Cutting-Plane Algorithm We start with a subset S0 ⊆ V of some ground set V with a reasonably good lower bound γ = f(S0) · maxff(S): S ⊆ V g. Using this information, we cut off the feasible sets on which the objective function values are guaranteed to be not better than f(S0). In this section, we address a method for solving the submodular maximization problem (1) based on this idea along the line of cutting-plane methods, as described by Tuy [23] (see also [6, 7]) and often successfully used in algorithms for solving mathematical programming problems [18, 11, 20]. 2.1 Lovasz´ extension For dealing with the submodular maximization problem (1) in a way analogous to the continuous counterpart, i.e., convex maximization, we briefly describe an useful extension to submodular func- tions, called the Lovasz´ extension [15, 16]. The relationship between the discrete and the continuous, described in this subsection, is summarized in Table 1. 1A submodular function is called nondecreasing if f(A) · f(B) for (A ⊆ B). For example, an entropy function is nondecreasing but a cut function on nodes is not. 2 c* Table 1: Correspondence between continu- H+ ous and discrete. H* H P H - (discrete) (continuous) y1 V ! R Eq.) (2) ^ Rn ! R f : 2 = f : d1 y ⊆ Eq.() (3) 2 Rn v 2 S V IS d2 f is submodular Thm.() 1 f^ is convex Figure 1: Illustration of cutting plane H. For H¤ and c¤, see Section 3.2. n Given any real vector p 2 R , we denote the m distinct elements of p by p^1 > p^2 > ¢ ¢ ¢ > p^m. Then, the Lovasz´ extension f^ : Rn ! R corresponding to a general set function f : 2V ! R, which is not necessarily submodular, is defined as P ^ m¡1 ¡ f(p) = k=1 (^pk p^k+1)f(Uk) +p ^mf(Um); (2) ^ where Uk = fi 2 V : pi ¸ p^kg. From the definition, f is a piecewise linear (i.e., polyhedral) func- tion.2 In general, f^ is not convex. However, the following relationship between the submodularity of f and the convexity of f^ is given [15, 16]: Theorem 1 For a set function f : 2V ! R and its Lovasz´ extension f^ : Rn ! R, f is submodular if and only if f^ is convex. P 2 f gn Now, we define IS 0; 1 as IS = i2S ei, where ei is the i-th unit vector. Obviously, there is 3 a one-to-one correspondence between IS and S. IS is called the characteristic vector of S. Then, the Lovasz´ extension f^is a natural extension of f in the sense that it satisfies the following [15, 16]: ^ f(IS) = f(S)(S ⊆ V ): (3) In what follows, we assume that f is submodular. Now we introduce a continuous relaxation of the problem (1) using the Lovasz´ extension f^. A polytope P ⊆ Rn is a bounded intersection of a finite f 2 Rn > · ¢ ¢ ¢ g set of half-spaces — that is, P is of the form P = x : Aj x bj; j = 1; ; m , where Aj is a real vector and bj is a real scalar.

View Full Text

Details

  • File Type
    pdf
  • Upload Time
    -
  • Content Languages
    English
  • Upload User
    Anonymous/Not logged-in
  • File Pages
    9 Page
  • File Size
    -

Download

Channel Download Status
Express Download Enable

Copyright

We respect the copyrights and intellectual property rights of all users. All uploaded documents are either original works of the uploader or authorized works of the rightful owners.

  • Not to be reproduced or distributed without explicit permission.
  • Not used for commercial purposes outside of approved use cases.
  • Not used to infringe on the rights of the original creators.
  • If you believe any content infringes your copyright, please contact us immediately.

Support

For help with questions, suggestions, or problems, please contact us