A Nonlinear Knapsack Problem

Home , Knapsack problem, Optimization problem

ELSEVIER Operations Research Letters 17 (1995) 103-110

A nonlinear Knapsack problem

Dorit S. Hochbaum School of Business Administration and Department of 1EOR, University of California, Berkeley, CA 94720, USA Received 1 February 1994; revised 1 May 1994

To the memory of Eugene Lawler

Abstract

The nonlinear Knapsack problem is to maximize a separable concave objective function, subject to a single "packing" constraint, in nonnegative variables. We consider this problem in integer and continuous variables, and also when the packing constraint is convex. Although the nonlinear Knapsack problem appears difficult in comparison with the linear Knapsack problem, we prove that its complexity is similar. We demonstrate for the nonlinear Knapsack problem in n integer variables and knapsack volume limit B, a fully polynomial approximation scheme with running time ()((1/e 2) (n + l/e2)) (omitting polylog terms); and for the continuous case an algorithm delivering an e-accurate solution in O(nlog(B/~)) operations.

Keywords: Convex optimization; Fully polynomial approximation scheme; Knapsack problem; Quadratic Knapsack problem; Allocation problem

O. Introduction In its most general form considered here the nonlinear Knapsack problem has the objective separable The nonlinear Knapsack problem (NLK) is concave and the packing constraint separable con- a generalization of the well-known integer Knap- vex: sack problem which maximizes a linear objective function representing utilities associated with Max ~ fj(xj) choosing items (the number of units of item j is j=l represented by the variable x j) subject to a "packing" constraint: s.t. ~ gj(xj) ~< B, (NLK) J max pixj ajxj <~ B, uj >1 xj >i 0 and O~

x~ integer forj = 1 .... , n~. The functions J~ are assumed concave and nonde- ) creasing, and the functions gj are assumed convex

and nondecreasing. Without loss of generality, branch-and-bound algorithm is based on solving B and uj are integers. the continuous relaxation first. For solving the con- There are several complexity issues that arise tinuous problem, they rely on the availability of when the functions fj and g~ take real values on derivatives in analytic form, and the solution of integer inputs. In this case, the computation model analytic equations. Since solving the continuous assumed is the unit cost model, i.e. any arithmetic relaxation is itself a challenging problem in their operation or comparison is counted as a single approach, the algorithm is only applicable to operation, even if the operands are real numbers. limited cases of the nonlinear Knapsack problem. Another issue is the interpretation of what it means The results reported in this paper indicate that to solve a problem in reals. Such issues are investi- the complexity of the nonlinear Knapsack problem gated in Section 2. is quite close to that of the linear one. We consider Well studied classes of (NLK) are the quadratic solving optimally the continuous nonlinear Knap- Knapsack with the functions fj quadratic and con- sack problem and present a fully polynomial ap- vex and the functions g~ linear, and the nonlinear proximation scheme for the (integer) nonlinear Knapsack problem in which the functions g~ are Knapsack. linear. Also the continuous version of the problem We use the concept of e-accuracy to solve the is important and comes up in applications such as continuous problem. For a prespecified accuracy of production planning [18], marketing [12], capital e, a solution is said to be e-accurate if it is at most at budgeting [14] and numerous others. a distance of e (in the L~ norm) from an optimal There is an extensive literature on solution solution. In other words, it is identical to the opti- methods for the nonlinear Knapsack problem and mum in O(log I/e) significant digits. The continu- its special cases, and the few mentioned here form ous problem is shown to be solvable with an e- only a partial list. This literature focuses on cases accurate solution in time O(n log B/e). This running where the functions are analytic and differentiable. time is not possible to improve as it is equal to the For the integer problem, Marsten and Morin [13] running time for solving the continuous allocation developed a branch-and-bound approach com- problem, which is a simple special case of the non- bined with a dynamic programming approach de- linear Knapsack problem. In [6] it is proved that veloped earlier in [16]. Mathur et al. [14] also this running time for the allocation problem is developed a branch-and-bound approach. They optimal for the comparison model in that it meets noted the representation of the objective function the established lower bound, and cannot be sub- as a piecewise linear function, but as they were not stantially reduced even for the algebraic tree model aware of an efficient approach to exploit this struc- that permits arithmetic operations in addition to ture they dismiss this approach as unlikely to lead comparisons and branchings and the floor opera- to improved algorithms. This representation is in tion. fact the key to the algorithms devised here. Using it, Finally, we show an adaptation of the fully poly- we establish that the continuous solution to the nomial approximation scheme for the linear integer nonlinear Knapsack problem is solvable efficiently Knapsack problem, to the nonlinear case as well, and in polynomial time. with similar complexity. Ziegler described in [18] an application with We use here a terminology that is common in the a specific objective function and packing con- context of the Knapsack problem. An item is said straint. He described for this problem a fully poly- to be selected if the corresponding variable is set to nomial approximation scheme. We demonstrate one. The objective function coefficients are referred that such an approximation scheme exists for all to as profits, and the single constraint as the pack- instances of (NLK), and that Ziegler's scheme re- ing constraint. The coefficients in the packing con- sults as a corrollary of the general case. straint are referred to as weights, and the value of Recently, Bretthauer and Shetty [1, 2] described the right-hand side of the constraint, as the Knap- a branch-and-bound algorithm for the quadratic sack volume. We use boldface letters to denote case [2], and for the nonlinear case [1]. The vectors, N'N~ denotes the maximum norm and D.S. Hochbaum/ OperationsResearch Letters 17 (1995) 103-110 105 e denotes the vector of all l's (1 .... ,1) of compat- The complexity of solving this 0/1 Knapsack prob- ible dimension. All logarithms are base 2. lem is O(Nmin{B, P*}) for P* denoting the opti- In Section 1 we describe the piecewise linear mal solution value. For this dynamic programming approximation that is used for the algorithms de- to work, it is necessary that the objective function vised here. Section 2 presents a polynomial algo- and constraint coefficients are integral. The algorithm for the continuous nonlinear Knapsack rithm runs in O(NB) operations if only the con- problem, and in Section 3 we provide the descrip- straint coefficients (the weights) are integral. It runs tion of the fully polynomial approximation scheme. in time O(NP*) if only the objective function coefficients are integral. The O(NP*) dynamic programming algorithm is given in Section 3.1. 1. The piecewise linear approximation If the functions gj map integers to integers or alternatively the functions fj map integers to inte- A piecewise linear approximation of the func- gers, then (NLK) is solvable in O(BY~= l uj) steps tions fj and the functions #j is used to convert the (or O(P* y~.= 1 uj) steps), which is the same complex- nonlinear Knapsack problem (NLK) into a 0/1 ity as the corresponding linear Knapsack problem. Knapsack problem. The integer (NLK) is equiva- lent to the problem, (PLK), derived by a piecewise linear approximation on the integer grid. This 2. Solving the continuous nonlinear Knapsack is achieved by replacing each variable xj by the problem sum of binary variables Y~I xlj, and letting plj =fj(i) -fj(i - 1), and alj = 9j(i) - 9j(i - 1): The notion of solving a continuous problem is uj not well defined for nonlinear problems. The con- Max ~ • pijxij tinuous problem has only been solved in special /=1 i=t cases when the derivatives of the functionsf~ and 9j exist. Even then it may be impossible to state the s.t. ~ ~ aijxij ~ B, (PLK) optimal solution accurately. The only case in which j=l i=1 an exact and accurate solution can be obtained is xij~{O, 1}, i=l,...,uj, j=l,...,n. the quadratic convex optimization problem over linear or convex quadratic constraints (see, for in- (PLK) is an ordinary 0/1 Knapsack problem, ex- stance, [ 10]). cept that the coefficients Pij or aij may not be It is not surprising that difficulties are encoun- integers or rationals. In order for this formulation tered in the continuous version since the solution to be valid for (NLK) the variables in a solution could require infinite representation. For example, vector x~j must satisfy for each given j, that when let a simple nonlinear Knapsack problem be given ordered as an array (xii .... , x.j,j), two l's may not with fl(xl) = 6xl - x3,fz(x2) = 0, the packing be separated by a 0, or two O's by a 1. constraint xl + x2 ~< 2, and the variables are bi- It is easy to see that the concavity of fj and the nary. The optimal solution to this 2 variable prob- convexity of 9j guarantee this condition. It follows lem is (x/2, 2 - x/~, which is irrational. The solu- that when a o- and P~i are integers, then techniques tion may even lack an algebraic representation, that are used for the 0/1 Knapsack problem are hence the output does not have a finite length. So applicable here as well. There is a known dynamic solving a nonlinear problem is challenging even programming algorithm for the 0/1 Knapsack when the nonlinearities are as simple as poly- problem: nomials. We therefore use the practical notion of N ~-accurate solution (previously used in [7]) to rep- max t N piXj aix j <~ B, xj >1 0 and resent the continuous solution to the problem. Lj=I j=l A solution, x ~ is e-accurate if there exists an optimal solution to the continuous problem, x* such xj integer for j = 1 .... , N}. that IIx~ -x* II~ <~ ~. That is, e is the accuracy 106 D.S. Hochbaum / Operations Research Letters 17 (1995) 103.- 110

required of the solution vector produced, and j = 1 .... , n, the allocation problem (SAP) is equiva- [-log l/e7 is the number of significant digits. lent to finding the largest B entries in the arrays. One Using the notion of e-accuracy, it is possible to important property of the allocation problem that solve some classes of constrained nonlinear optim- follows, is that a greedy algorithm, selecting one ization in polynomial time [7]. Hochbaum and entry at a time according to largest entry value Shanthikumar [7] obtained polynomial-time algo- (function increment), until B entries are selected, rithms by using "proximity" results on the distance delivers an optimal solution. Although this greedy between optimal scaled solutions and the continu- algorithm is not polynomial, we shall make use of the ous optimal solution. (There is also a proximity implied property in the derivation of a fully poly- result on the distance with respect to the integer nomial approximation scheme for the integer (NLK). optimal solution, which is not relevant for the non- Due to the concavity of the functions fj, the linear Knapsack problem.) Hochbaum, in [6], chosen entries form a consecutive array of values solved various allocation problems to e-accuracy, and the last entry selected in array j corresponds to using a different and tighter proximity theorem, the integer value of the variable x j, If there are less applicable for allocation problems. than B nonnegative values in the arrays, then the The simple allocation problem is closely related to solution to the (SAP) consists of all the entries with the Knapsack problem. It differs from it in that the positive value. packing constraint is simpler - a sum of all vari- Hochbaum showed in [6] how to obtain an ~- ables, whereas for the Knapsack problem it is accurate solution to the continuous relaxation of a weighted sum of the variables. The algorithms we the allocation problem (SAP) in O(n log B/e) steps. develop make use of analysis done for the alloca- This solution is obtained by solving using greedy tion problem and its continuous relaxation. The the integer problem on a scaled grid and then formulation of the simple allocation problem (SAP) relying on a proximity theorem. The scaled prob- is as follows: lem is: .L (SAP) max ~ Ji(xj) max L fj(s. xj) j~l j=l B • xj<~B, ~ xj<<.- j=l j=l S Uj 0 <~ xj <~ uj integer j = 1 ..... n. O<~xj<~--integer j= 1 ..... n. s The extensive literature on the allocation problem and its extensions is surveyed in a book by Ibaraki Let the solution to the scaled problem be denoted and Katoh [9]. by x ~s), and an optimal solution to the continuous Frederickson and Johnson [5] (see also [6] ) gave problem x*. The proximity result states that for the most efficient algorithm known for the simple any x t~) there exists an optimal continuous solution allocation problem. The running time of this algo- x* so that:x* >7 x ~s) - s.c. As a corollary, since the rithm is O(n log B/n). It was proved in [6] that this solution x ~) satisfies x ¢~. e = B, running time is optimal with respect to the com- ilx* -x~'~ll~ <~ n.s. parison model and is close to the lower bound for the algebraic tree model. This running time de- The result is independent of the grid's uniformity. pends linearly on the number of variables, but only Given the problem (PLK) we can represent it as logarithmically on the value of B. an allocation array problem. We have n arrays, The allocation problem may be viewed as where each unit entry corresponding to the variable a problem defined on n arrays of length up to x~j is associated with a contribution of pij units to B each: Given n arrays of nonnegative entries of the objective and ai~ units to the constraint. We function increments, {fj(i)-fj(i-1)}~_ 1, for replace such entry by aii entries, each with the value D.S. Hochbaum / Operations Research Letters 17 (1995) 103 - 110 107

Pij/aij associated with it and a unit contribution to accurate is also optimal. The resulting quadratic the constraint. (If aij = 0 then we keep only one allocation problem is solved using the linear time such entry with the value Pij associated with it. algorithm in [4]. Such a case is unlikely, but if it occurs then the solution method is easier, as all such variables will be included.) We then solve the resulting allocation 3. A fully polynomial approximation scheme problem. This transformation corresponds to intro- ducing new variables, xij = ylj/aij and solving the A fully polynomial approximation scheme is integer allocation problem a family of approximation algorithms {A~}, where algorithm At is an e-approximation algorithm with Max relative error bounded by e for all possible problem j=l , =L, --a,3 Yij" instances. In addition, the running time of At depends polynomially on the input size and on s.t. ~. Yij = B, (*) I/e. j=l i=l A fully polynomial approximation scheme for 0 ~< Yij ~< ali, the 0/1 Knapsack problem and the integer Knap- yijinteger, i= 1 ..... uj j=l ..... n. sack problem was first devised by Ibarra and Kim [8] and later improved by Lawler [11]. Lawler Let the vectory* be the optimal solution to (,). Let further addressed the nonlinear Knapsack problem Ylj = (1/aOYi* for all i,j. The allocation proximity as well. The running time of his approximation theorem implies that scheme depends on ~". u~. This quantity is not 1 =1 polynomial in the length of the input and hence the IJx* -.¢11~ ~< n.max . scheme is not polynomial. We demonstrate here ij that Lawler's fully polynomial approximation In order to obtain the e-accuracy we modify the scheme for the linear Knapsack problem is imple- transformation of variables to, xij = yij/sij where mentable for (NLK) in O((1/e 2) (nlog B + log(l/e) sij = aJ n/e ] and duplicate each entry (i, j)sq times. log n + ( 1/e 2) log(l/e))). Although this increases the size of the arrays, it We first present, for the sake of completeness, does not cause an increase in the running time the simplest form of the fully polynomial ap- required to solved the allocation problem (,) as proximation scheme for the 0/1 Knapsack, that depends only on the number of arrays and the max{57~= lpJxjl •i=N 1 ajxj <. B, xj ~ {0, 1} for j = right-hand side value. The right-hand side is also 1 ..... N}. Following this presentation it is demon- scaled so that all coefficients are integers: strated how the operations of this approximation scheme are to be modified to achieve the same = Br,,/e-1. result for (NLK). Consequently, the running time is O(nlog B/n)= O(n log B/e). 3.1. An O(NP*) algorithm for solving the 0/1 Knap- In the quadratic Knapsack problem, the func- sack problem tionsfj are quadratic concave and 9~ are linear. The optimal solution in this case is of polynomial length The following dynamic programming algorithm in the size of the input, so for e polynomial in the solves the problem optimally for integer objective input size an e-accurate solution is also optimal. function coefficients: The quadratic continuous Knapsack is known to Let Fi(i) denote the smallest Knapsack volume that be solvable in linear time I-3]. The algorithm given yields an objective function value of i, involving in this section provides an alternative way for solv- variables in the set {1 ..... j}. The boundary condi- ing the continuous quadratic Knapsack. For the tions are specified accuracy e we duplicate each entry 1/eaij times, e is chosen so that any solution that is e- Fj(o) = o, j = 1 ..... N 108 D.£ Hochbaum / Operations Research Letters 17 (1995) 103-110

and the recursive formula is Let S* be the set of indices of the variables in the optimal solution to the Knapsack problem, and F~(i) = min{Fj l(i -- pj) q- aj, Fj_ l(i)}. S(k) the set of indices of the variables in the optimal Using the boundary conditions the table of values solution to the scaled problem. is evaluated in increasing order of objective func- Z PJ>I Z k[-pHkJ>l ~ kLp~/k-J tion values: j E S(k) j ~ S(k) j E S* FI(1),F2(1),...,FN(1); F1(2).... ,FN(2); .... >~ ~ (pi-k)>~ ~ pj-klS*[. (3.2) jeS* jeS* The computation terminates once the largest value of i is found so that FN(i) <<, B. Each function evalu- Hence, the absolute error of the "scaled" solution is ation is done in O(1) steps, and there are a total of at most k IS*h, and the relative error (using the O(NP*) function evaluations, where P* is the opti- lower bound in (3.1)) is mal value of the objective function. The running klS*l k time of this dynamic programming algorithm is s= ~

For a chosen value of e, Lawler [11] defined 3.3. A fully polynomial approximation schemefor the large items as those with p~ t> T = 21 ePo, and small nonlinear Knapsack items are all the others. The scaling factor selected is k = z1 e2 Po. The reduced running time is achieved It remains to show how to implement each one of by running the dynamic programming algorithm the steps of the approximation algorithm A~ so that only for the large items. The small items are then it applies to the nonlinear Knapsack problem in packed in order of nonincreasing pHaj ratio while polynomial time: To find the value of Po, we repres- there is still slack in the packing constraint. The ent (NLK) as the allocation problem (,) in the total error resulting from the union of these two variables Yij = aijx~j. Recall the greedy property of sets of selected variables is at most e. To verify that the allocation problem. For all variables Yij for the total error does not exceed e, note that the total a fixed (i,j), the objective coefficients are the same. error derived from the dynamic programming solu- Hence, it is always possible to find one optimal tion on the large items does not exceed kP*/T as solution in which at most one value ofx~j = y~/aq is the number of large items in the optimal solution to not an integer, but rather a fraction in the interval the scaled problem is P*/T. For the small items, the (0, 1). From the greedy property, this variable has error does not exceed the value of T. The sum of the the next largest ratio coefficient pi~/a~j among all relative errors is hence unselected items, and the smallest among all se-

1 2 lected ones. Hence, when we round it down to 0, we _k T ~e Po +½~Po = get precisely the set of] items of largest ratio. The T+P---~< ~e~o Po ~" maximum of Pmax and the objective function value The computational advantage is derived from the corresponding to this rounded down solution, is fact that the total number of large items that needs the value of Po- to be considered is small. The number of different The computationally most expensive step is step values of pj(k) is bounded by P*/k. With the values 2. There are up to 8/e 2 different profits in the scaled of k and T as above, the number of different values problem. For each, we need to identify the nj ones is bounded by 8/~ 2 . As for each value of pj(k), there with the least weight, and provide a pointer in each could be at most nj = L(P*/k)/p~(k)J items of size array to the index of the first such item. We first p~(k) in an optimal solution, and among all those of compute for each of the 8/e 2 arrays the position of the same scaled profit, those with the smallest the pointers. We proceed by computing those weights would be selected. Other large items need pointers, one at a time in decreasing value of scaled not be considered as they are dominated. With this profits. type of argument, Lawler proved that the number Let the current scaled profit for which we search of large items to be considered is (6/e 2) log2(4/e). for the pointers be q. Consider the n arrays of scaled Consequently, the running time required to solve integer profits. In each array conduct binary search the dynamic programming algorithm that runs on to determine the first entry of value q or less. Hence, large items is O((1/e 4) log2(1/e)). for each value of q, q = 1, ..., 8/e 2 the running time To summarize, the steps of the approximation is O(nlogB) for a total of O((1/eZ)nlogB) steps. algorithm A~ for the 0/1 linear Knapsack problem Once these pointers are identified we scan the ar- are as follows: rays for each value of scaled profit q and record the up to n~ items with smallest weight and profit q. The 1. Find the value and the set of elements corres- amount of computation required is the total ponding to Po. number of large items O((1/e2)log(1/e)) multiplied 2. Find the "large" items that are candidates for by a factor of logn to maintain a sorted array of inclusion in the optimal solution. nonincreasing weights. The running time is there- 3. Solve for the "large" items the scaled problem, fore O((1//3 2) 1og2(1/e) log n). using dynamic programming. Finally, to pack the largest number of largest 4. Find the largest ratio "small" items that can be ratio small items, as in step 4, repeat the same packed in the remaining volume of the Knapsack. procedure as for step l, and solve the allocation 110 D.S. Hochbaum / Operations Research Letters 17 (1995) 103 110

problem (,) for the small items, rounding down the [7] D.S. Hochbaum and J.G. Shanthikumar, "Convex one fractional variable resulting in the optimal separable optimization is not much harder than linear solution. The running time of steps 1 and 4 is optimization", J. Assoc. Comput. Mach. 37, 843-862 (1990). dominated by that of steps 2 and 3. Steps 1 and [8] O.H. Ibarra and C.E. Kim, "Fast approximation algo- 4 require O(nlog(B/n)) operations each. Step rithms for the Knapsack and sum of subset problems", J. 2 requires O(n(1/e2)logB + (1/~2)log(1/e)logn) Assoc. Comput. Mach. 22, 463-468 (1975). operations. Step 3 requires O((1/e 4) lOgE(1/e)) [9] T. lbaraki and N. Katoh, Resource Allocation Problems: operations. The total computation time is hence Algorithmic Approaches, MIT Press, New York, 1988. [10] MK. Kozlov, S.P. Tarasov and L.G. Khachian, "Poly- O((l/e 2) (n log B + log(l/t) log n + (1/E2) log (l/t))). nomial solvability of convex quadratic programming", For e chosen small enough this running time is the Doklady Akad. Nauk SSR 5, 1051-1053 (1979). [Translated same as the time required to solve the approxima- in Soy. Math. Doklady 20, 1108-1111 (1979). tion scheme for the 0/1 Knapsack problem and is [11] E. Lawler, "Fast approximation algorithms for Knapsack hence of similar complexity. problems", Math. Oper. Res. 4, 339-356 (1979). [12] H. Luss and S.K. Gupta, "Allocation of effort resources among competing activities", Oper. Res. 23, 360-366 (1975). References [13] R.E. Marsten and T.L. Morin, "A hybrid approach to discrete mathematical programming", Math. Programming [1] K. Bretthauer and B. Shetty, "The nonlinear resource 14, 21-40 (1978). allocation problem", manuscript, Texas A&M University, [14] K. Mathur, H.M. Salkin and S. Morito, "A Branch and 1993. search algorithm for a class of nonlinear Knapsack prob- [2] K. Bretthauer, B. Shetty and S. Syam, "'A branch and lems", Oper. Res. Lett. 2, 155-160 (1983). bound algorithm for integer quadratic knapsack prob- [15] S. Martello and P. Toth, Knapsack Problems: Algorithms lems", Manuscript, Texas A&M University (1993), to ap- and Computer Implementations, Wiley, New York, pear in ORSA J. Computing. 1990. [3] P. Brucker, "An O(n) algorithm for quadratic Knapsack [16] T.L. Morin and R.E. Marsten, "An algorithm for nonlinear problems", Oper. Res. Lett. 3, 163-166 (1984). Knapsack problems", Management Sci. 22, 1147-1158 [4] S. Cosares and D.S. Hochbaum, "A strongly polynomial (1976). algorithm for the quadratic transportation problem with [17] E. Tardos, "A strongly polynomial algorithm to solve fixed number of suppliers", Math. Oper. Res. 19, 94-111 combinatorial linear programs", Oper. Res. 34, 250-256 (1994). (1986). [5] G.N. Frederickson and D.B. Johnson, "'The complexity of [ 18] H. Ziegler, "Solving certain singly constrained convex op- selection and ranking in X + Y and matrices with sorted timization problems in production planning", Oper. Res. columns", J. Comput. Systems Sci. 24, 197-208 (1982). Lett. 1,246-252 (1982). [6] D.S. Hochbaum, "Lower and upper bounds for the allocation problem and other nonlinear optimization problems", Math. Oper. Res. 19, 390-409 (1994).