TMA521/MMA511 Large-Scale Optimization Lecture 1 Introduction: simple/difficult problems, matroid problems

Ann-Brith Str¨omberg Professor of Mathematical Optimization

2020–11–02

1 / 28 TMA521/MMA511 Optimization, project course

◮ Examiner, lecturer & course responsible Ann-Brith Str¨omberg ([email protected]) ◮ Teaching assistant Sunney Fotedar ([email protected]) ◮ Schedule ◮ 13–14 lectures ◮ 2 workshops (mandatory presence; optional dates) ◮ 1 student presentation + review (mandatory presence; optional dates) ◮ Linux computer labs (remote access) booked 2–4 hours per week for project work. Advisement in Zoom. ◮ Course homepage in Canvas ◮ Two projects, including written reports ◮ Lagrangean relaxation for a VLSI design problem (Matlab) ◮ Modelling and decomposition of a combinatorial optimization problem. Algorithm design.

2 / 28 Requirements for passing the course and examination

1. Each student must take active part in two scheduled workshops, and in the two project assignments 2. The findings of your projects are to be presented in the form of two written reports (plus one pre-report within the second project) 3. Each group shall present project 1 at a seminar, and act as opponents at another group’s presentation of the same project 4. A written exam 5. Each student must take active part in all of these activities 6. Project groups consist of two–three students

3 / 28 Some practicalities about the workshops and presentations

◮ Each of the two workshop sessions will be split into 3–4 occasions (see detailed course plan) ◮ Each student/group: sign up for one occasion/workshop (w46 & w49)

◮ The presentation/review session will be split into 3–4 occasions (see detailed course plan) ◮ Each project group: sign up for one occasion (w48)

◮ Join a project group in Canvas (not later than Thursday this week)

◮ The hours may be slightly adjusted

◮ Doodles/choodles for signing up will be posted in Canvas

4 / 28 Course evaluation

◮ Questionnaires will be sent out to all students registered for the course after the exam week ◮ The examiner calls the student representatives to the introductory, (recommended) middle, and final meetings

Student representatives (Chalmers): ◮ Jakob Hend´en (MPDSC) ◮ Vincent Molin (MPENM) ◮ Simon Pleijel (MPALG) ◮ Hanna Skytt (MPENM) ◮ Samuel Thor´en (MPENM)

◮ Any voluntary GU student?

5 / 28 Course literature

◮ The compendium “Lagrangean duals and subgradient methods for linear and mixed-binary linear optimization”, Str¨omberg (2019)

◮ The book chapter “A primer in column generation”, Desrosiers and Lubbecke¨ (2005)

◮ Articles and book chapters (pdf’s and links in Canvas)

◮ Lecture notes (published in Canvas)

6 / 28 Aim

◮ Purpose of the course: provide an overview of the most important principles for the efficient solution of practical large-scale optimization problems, from theoretical foundations via modelling to method implementations

◮ By means of ◮ a series of lectures ◮ workshops/modelling exercises ◮ project work, in which the knowledge gained is applied to model, decompose, and efficiently solve some relevant large-scale optimization problems ◮ report writing

7 / 28 Learning outcome

After completion of this course, the student should

◮ be able to independently analyze and suggest modelling and solution principles for large-scale complex optimization problems, and

◮ have sufficient knowledge to use these principles successfully in practice through the use of computation software for optimization problems.

8 / 28 Content

◮ Most large-scale optimization problems have inherent structures that can be exploited to solve them efficiently ◮ The course deals with some principles through which large-scale optimization problems can be attacked ◮ Techniques: decomposition–coordination (or, distributed algorithm–consensus). Based on convexity theory, polyhedral theory, integer optimization and duality theory. ◮ The practical moments: ◮ An exercise/workshop on modelling and solution of a design problem (Wednesday or Thursday, November 11–12) ◮ Two project assignments (including the second workshop, on December 2–3), where large-scale optimization problems are modelled and solved using theory and techniques from the literature and presented during the lectures

9 / 28 Content, cont’d.

◮ Problems: Complexity, simple/difficult optimization problems, integer linear optimization problems, unimodularity, convexity.

◮ Principles: Decomposition–coordination, restriction, relaxation, bounds on the optimal value, projection, variable fixing, dualization, neighbourhoods, heuristics, local search methods.

◮ Methods: Lagrangean duality, subgradient methods, (ergodic) convergence, recovery of integer solutions, Lagrangean heuristics, cutting planes, column generation, coordinating master problem, Dantzig–Wolfe decomposition, Benders decomposition.

10 / 28 General topic: Turn difficult problems into sequences of simpler ones using decomposition and coordination

Prerequisites ◮ Linear Optimization (, LP) [Mixed] Integer Linear Optimization ([M]ILP) Non-Linear Optimization (NLP)

Decomposition methods covered ◮ Lagrangean relaxation of LP as well as Mixed Binary Linear Optimization (MBLP) ◮ Subgradient optimization (for Lagrangean duals of LP/MBLP) ◮ Heuristics and Greedy Algorithms (for MBLP and matroids) ◮ Dantzig–Wolfe decomposition (for LP) ◮ Column generation (for LP and MBLP) ◮ Benders decomposition (for MILP, NLP)

11 / 28 Important properties of simple problems

◮ What we here call simple problems can be solved in polynomial time with respect to the problem size (i.e., numbers of variables, constraints, ...)

◮ For simple problems, there exist polynomial algorithms preferably with a small largest exponent

◮ E.g., sorting an array of n elements can be done in time proportional to at most (i.e., worst case performance)

◮ n2 operations (bubble sort)

◮ n log n operations (heapsort, quicksort)

12 / 28 Examples of simple problems (Wolsey, 1998)

◮ Network flow problems:

◮ Shortest paths ◮ Maximum flows ◮ Minimum cost (single-commodity) network flows ◮ Transportation and assignment problems ◮ Maximum cardinality/weight matchings

◮ Problems over simple matroids

◮ E.g., spanning trees and forests

◮ Linear optimization problems

13 / 28 Example: Shortest path

Find the shortest path from node 1 to node 7

d4=9 i = 2 5 d9=2 d =2 1 d3=1 d8=4 7 1 3

d6=2 d5=3 6 d10=8 d2=2 4 d7=5

di = length of edge i Shortest path from node 1 to node 7

d4=9 2 5 d9=2 d =2 1 d3=1 d8=4 7 1 3

d6=2 d5=3 6 d10=8 d2=2 4 d7=5

Total length: 12

14 / 28 Example: Maximum flow

Find the maximum flow from node 1 to node 7 A maximum flow from node 1 to node 7

k4=4 x4=3 2 5 2 5 k9=2 x9=2 k =6 x =6 1 k3=4 1 x3=3 k8=1 7 x8=1 7 1 3 1 3

k6=2 k5=3 6 k10=6 x6=0 x5=3 6 x10=5 k2=2 x2=1 4 k7=1 4 x7=1

ki = flow capacity of arc i (minimum arc capacity: 0) xi = optimal flow through arc i (≥ 0). Total flow: 7 A minimum cut separating nodes 1 and 7

k4−x4=1 2 5 k9−x9=0 k −x =0 1 1 k3−x3=1 k8−x8=0 7 1 3 k6−x6=2 k −x =0 5 5 6 k10−x10=1 k2−x2=1 4 k7−x7=0

ki − xi = residual flow capacity on arc i. Total cut capacity: k9 + k8 + k5 + k7 = 7

15 / 28 Matroids and the

◮ Greedy algorithm ◮ Create a “complete solution” by iteratively choosing the best alternative ◮ Never regret a previous choice

◮ Which problems can be solved using such a simple method?

◮ Problems whose feasible sets can be described by matroids

16 / 28 Example I: Matric matroids

◮ E = a set of column vectors in Rn

◮ F = the set of linearly independent subsets of vectors in E.

10210 ◮   Let n = 3 and E = [e1,..., e5]= 01100 32001

◮ We have:

◮ {e1, e2, e3}∈F and {e2, e3}∈F but

◮ {e1, e2, e3, e5} 6∈ F and {e1, e4, e5} 6∈ F

17 / 28 Example II: Graphic matroids

◮ E = {e1, e2, e3, e4, e5, e6, e7} = the set of edges in an undirected graph

◮ F = the set of all cycle-free subsets of edges in E

e2 e2 e2

e1 e3 e5 E = {e ,..., e } 1 7 e7 e7 e4 e4 e2 e6 e1 e3 e 5 e2 e2 e7 e6 e4 e1 e3 e1 e5 e7 e7 e4

◮ {e1, e2, e4, e7} ∈ F, {e2, e4, e7} ∈ F, {e2, e3, e5} 6∈ F, {e1, e2, e3, e7} ∈ F, {e1, e4, e5, e6, e7} 6∈ F, {e2} ∈ F

18 / 28 The greedy algorithm for minimization problems

1. A := ∅

2. Sort the elements of E in increasing order with respect to the costs w(e), e ∈E

3. Take the first element e ∈E in the list. If A ∪ {e} is still independent =⇒ let A := A ∪ {e} and let E := E \ {e}

4. Repeat from step 3. with the next element—until either E = ∅ or |A| is maximal

19 / 28 Matroids and the greedy algorithm applied to Example II

◮ Let w(e) be the cost of element e ∈E. Problem: Find the element I ∈ F of maximal cardinality such that the total cost is at minimum

◮ Example II, continued: w(E) = (7, 4, 2, 15, 6, 3, 2)

4 3 3 7 2 6 7 2 2 2 15

An element I ∈ F of maximal cardinality with minimum total cost

20 / 28 Example II: minimum spanning trees (MST)—graphic matroids

◮ The maximal cycle-free set of links in an undirected graph is a spanning tree ◮ In a graph G = (N , E), a spanning tree has |N|− 1 links ◮ Classic greedy algorithm—Kruskal’s algorithm has complexity O(|E| · log(|N |)). The main complexity lies in the sorting itself ◮ Prim’s algorithm builds the spanning tree through graph search techniques, from node to node; complexity O(|N |2) 4 3 3 2 6 5 1

21 / 28 Example III: The traveling salesperson problem (TSP)

The greedy algorithm for the TSP: 1. Start in node 1 2. Go to the nearest node which is not yet visited 3. Repeat step 2 until no nodes are left 4. Return to node 1; the tour is closed

◮ Greedy solution Not optimal whenever c > 4 2 2 1 1 1 1 1 2 3 1 2 3 2 2 2 1 2 2 2 1 5 5 4 c 4 c Optimal solution for c ≥ 4

22 / 28 Example IV: the (SPP)

◮ The greedy algorithm constructs a path that uses – locally – the cheapest link to reach a new node. Optimal? ◮ Greedy solution Not optimal whenever c > 9

1 1 5 5 5 5

s t s t

1 c 1 c 2 2

Optimal solution for c ≥ 9

23 / 28 Example V: Semi-assignment (Partition matroid)

n n minimize f (x) := w x , X X ij ij i=1 j=1 n subject to x = 1, i = 1,..., n, X ij j=1

xij ∈ {0, 1}, i = 1,..., n, j = 1,..., n.

◮ Algorithm For each i:

1. choose the best (lowest) wij

2. Set xij = 1 for that j, and xij = 0 for every other j

24 / 28 Problems over matroid intersections

◮ Given two matroids M = (E, P) and N = (E, R), find the maximum cardinality set in P ∩ R

◮ Example 1: The assignment problem forms the intersection of two partition matroids. Draw illustration!

◮ The intersection of two matroids can not generally be solved by using the greedy algorithm

◮ There exist polynomial algorithms for them, though ◮ Examples: bipartite matching and assignment problems can be solved as maximum flow problems, which are polynomially solvable

25 / 28 Problems over matroid intersections, cont.

◮ Example 2: The traveling salesperson problem (TSP) is the intersection of three matroids: ◮ two partition matroids ◮ one graph matroid (formulation on next page: assignment + tree constraints)

◮ TSP is not solvable in polynomial time, as far as we know so far, i.e., as long as P 6= NP

◮ The TSP—different mathematical formulations give rise to different algorithms when Lagrangean relaxed or otherwise decomposed

26 / 28 TSP: Assignment formulation for directed graphs

minimize c x , X X ij ij i∈N j∈N \{i} subject to x = 1, i ∈N , (1) X ij j∈N \{i} x = 1, j ∈N , (2) X ij i∈N \{j} x = |N |, (3) X X ij i∈N j∈N \{i} x ≤ |S|− 1, ∅= 6 S⊂N , (4) X X ij i∈S j∈S\{i} xij ∈ {0, 1}, i ∈N \{j}, j ∈N . (5) ◮ (1)–(2): assignment; (3): i (1) = j (2) (redundant when (1) or (2) is present); (4): cycle-freeP P ◮ Relax (3)–(4) ⇒ assignment; (5) can be relaxed: xij ≥ 0 (LP) ◮ Relax (1)–(2) ⇒ directed 1-tree (polynomially solvable) 27 / 28 Next lecture

◮ Structured optimization problem

◮ Lagrangean relaxation: relax complicating constraints + penalise the objective for violating constraints ⇒ decomposes into simpler subproblems ⇒ coordinate by optimizing the Lagrangean dual problem

◮ Theoretical properties; Lagrangean dual; primal and dual optimality conditions; solution methodology

28 / 28