CSC373 Review 373F19 - Nisarg Shah 1 BUT FIRST… Course Evaluation Low response rate 373F19 - Nisarg Shah 2 Complete your course evaluations… Check your e-mail for a link to your evaluations, or log-in to www.portal.utoronto.ca and click the Course Evals tab! 373F19 - Nisarg Shah 3 Should I expect drumrolls? Course Evaluation New response rate 373F19 - Nisarg Shah 4 Topics • Divide and conquer • Greedy algorithms • Dynamic programming • Network flow • Linear programming • Complexity • Approximation algorithms & local search • Randomized algorithms 373F19 - Nisarg Shah 5 Greedy Algorithms • Greedy algorithm outline ➢ We want to find the optimal solution maximizing some objective 푓 over a large space of feasible solutions ➢ Solution 푥 is composed of several parts (e.g. a set) ➢ Instead of directly computing 푥… o Consider one element at a time in some greedy ordering o Make a decision about that element before moving on to future elements (and without knowing what will happen for the future elements) 373F19 - Nisarg Shah 6 Greedy Algorithms • Proof of optimality outline ➢ Strategy 1: o 퐺푖 = greedy solution after 푖 steps o Show that ∀푖, there is some optimal solution 푂푃푇푖 s.t. 퐺푖 ⊆ 푂푃푇푖 • “Greedy solution is promising” o By induction o Then the final solution returned by greedy must be optimal ➢ Strategy 2: o Same as strategy 1, but more direct o Consider 푂푃푇 that matches greedy solution for as many steps as possible o If it doesn’t match in all steps, find another 푂푃푇 which matches for one more step (contradiction) 373F19 - Nisarg Shah 7 Dynamic Programming • Key steps in designing a DP algorithm ➢ “Generalize” the problem first o E.g. instead of computing edit distance between strings 푋 = 푥1, … , 푥푚 and 푌 = 푦1, … , 푦푛, we compute 퐸[푖, 푗] = edit distance between 푖-prefix of 푋 and 푗-prefix of 푌 for all (푖, 푗) o The right generalization is often obtained by looking at the structure of the “subproblem” which must be solved optimally to get an optimal solution to the overall problem ➢ Remember the difference between DP and divide-and- conquer ➢ Sometimes you can save quite a bit of space by only storing solutions to those subproblems that you need in the future 373F19 - Nisarg Shah 8 Dynamic Programming • Dynamic programming applies well to problems that have optimal substructure property ➢ Optimal solution to a problem contains (or can be computed easily given) optimal solution to subproblems. • Recall: divide-and-conquer also uses this property ➢ You can think of divide-and-conquer as a special case of dynamic programming, where the two (or more) subproblems you need to solve don’t “overlap” ➢ So there’s no need for memoization ➢ In dynamic programming, one of the subproblems may in turn require solution to the other subproblem… 373F19 - Nisarg Shah 9 Dynamic Programming • Top-Down may be preferred… ➢ …when not all sub-solutions need to be computed on some inputs ➢ …because one does not need to think of the “right order” in which to compute sub-solutions • Bottom-Up may be preferred… ➢ …when all sub-solutions will anyway need to be computed ➢ …because it is sometimes faster as it prevents recursive call overheads and unnecessary random memory accesses 373F19 - Nisarg Shah 10 Network Flow • Input ➢ A directed graph 퐺 = (푉, 퐸) ➢ Edge capacities 푐 ∶ 퐸 → ℝ≥0 ➢ Source node 푠, target node 푡 • Output ➢ Maximum “flow” from 푠 to 푡 373F19 - Nisarg Shah 11 Ford-Fulkerson Algorithm MaxFlow(퐺): // initialize: Set 푓 푒 = 0 for all 푒 in 퐺 // while there is an 푠-푡 path in 퐺푓: While 푃 = FindPath(s, t,Residual(퐺, 푓))!=None: 푓 = Augment(푓, 푃) UpdateResidual(퐺,푓) EndWhile Return 푓 373F19 - Nisarg Shah 12 Max Flow - Min Cut • Theorem: In any graph, the value of the maximum flow is equal to the capacity of the minimum cut. • Ford-Fulkerson can be used to find the min cut ∗ ➢ Find the max flow 푓 ∗ ➢ Let 퐴 = set of all nodes reachable from 푠 in residual graph 퐺푓∗ o Easy to compute using BFS ∗ ∗ ➢ Then (퐴 , 푉 ∖ 퐴 ) is min cut 373F19 - Nisarg Shah 13 LP, Standard Formulation 푛 푚 • Input: 푐, 푎1, 푎2, … , 푎푚 ∈ ℝ , 푏 ∈ ℝ ➢ There are 푛 variables and 푚 constraints • Goal: 373F19 - Nisarg Shah 14 LP, Standard Matrix Form 푛 푚 • Input: 푐, 푎1, 푎2, … , 푎푚 ∈ ℝ , 푏 ∈ ℝ ➢ There are 푛 variables and 푚 constraints • Goal: 373F19 - Nisarg Shah 15 Convert to Standard Form • What if the LP is not in standard form? ➢ Constraints that use ≥ o 푎푇푥 ≥ 푏 ⇔ −푎푇푥 ≤ −푏 ➢ Constraints that use equality o 푎푇푥 = 푏 ⇔ 푎푇푥 ≤ 푏, 푎푇푥 ≥ 푏 ➢ Objective function is a minimization o Minimize 푐푇푥 ⇔ Maximize −푐푇푥 ➢ Variable is unconstrained o 푥 with no constraint ⇔ Replace 푥 by two variables 푥′and 푥′′, replace every occurrence of 푥 with 푥′ − 푥′′, and add constraints 푥′ ≥ 0, 푥′′ ≥ 0 373F19 - Nisarg Shah 16 Duality Primal LP Dual LP • Weak duality theorem: 푇 푇 ➢ For any primal feasible 푥 and dual feasible 푦, 푐 푥 ≤ 푦 푏 • Strong duality theorem: ∗ ∗ 푇 ∗ ∗ 푇 ➢ For any primal optimal 푥 and dual optimal 푦 , 푐 푥 = 푦 푏 373F19 - Nisarg Shah 17 P • P (polynomial time) ➢ The class of all decision problems computable by a TM in polynomial time 373F19 - Nisarg Shah 18 NP • NP (nondeterministic polynomial time) ➢ The class of all decision problems for which a YES answer can be verified by a TM in polynomial time given polynomial length “advice” or “witness”. ➢ There is a polynomial-time verifier TM 푉 and another polynomial 푝 such that o For all YES inputs 푥, there exists 푦 with 푦 = 푝 푥 on which 푉(푥, 푦) returns YES o For all NO inputs 푥, 푉(푥, 푦) returns NO for every 푦 ➢ Informally: “Whenever the answer is YES, there’s a short proof of it.” 373F19 - Nisarg Shah 19 co-NP • co-NP ➢ Same as NP, except whenever the answer is NO, we want there to be a short proof of it 373F19 - Nisarg Shah 20 Reductions • Problem 퐴 is p-reducible to problem 퐵 if an “oracle” (subrouting) for 퐵 can be used to efficiently solve 퐴 ➢ You can solve 퐴 by making polynomially many calls to the oracle and doing additional polynomial computation 373F19 - Nisarg Shah 21 NP-completeness • NP-completeness ➢ A problem 퐵 is NP-complete if it is in NP and every problem 퐴 in NP is p-reducible to 퐵 ➢ Hardest problems in NP ➢ If one of them can be solved efficiently, every problem in NP can be solved efficiently, implying P=NP • Observation: ➢ If 퐴 is in NP, and some NP-complete problem 퐵 is p- reducible to 퐴, then 퐴 is NP-complete too o “If I could solve A, then I could solve B, then I could solve any problem in NP” 373F19 - Nisarg Shah 22 Review of Reductions • If you want to show that problem B is NP-complete • Step 1: Show that B is in NP ➢ Some polynomial-size advice should be sufficient to verify a YES instance in polynomial time ➢ No advice should work for a NO instance ➢ Usually, the solution of the “search version” of the problem works o But sometimes, the advice can be non-trivial o For example, to check LP optimality, one possible advice is the values of both primal and dual variables, as we saw in the last lecture 373F19 - Nisarg Shah 23 Review of Reductions • If you want to show that problem B is NP-complete • Step 2: Find a known NP-complete problem A and reduce it to B (i.e. show A ≤푝 B) ➢ This means taking an arbitrary instance of A, and solving it in polynomial time using an oracle for B o Caution 1: Remember the direction. You are “reducing known NP- complete problem to your current problem”. o Caution 2: The size of the B-instance you construct should be polynomial in the size of the original A-instance ➢ This would show that if B can be solved in polynomial time, then A can be as well ➢ Some reductions are trivial, some are notoriously tricky… 373F19 - Nisarg Shah 24 Approximation Algorithms • We focus on optimization problems ➢ Decision problem: “Is there…where…≥ 푘?” o E.g. “Is there an assignment which satisfies at least 푘 clauses of a given formula 휑?” ➢ Optimization problem: “Find…which maximizes…” o E.g. “Find an assignment which satisfies the maximum possible number of clauses from a given formula 휑.” ➢ Recall that if the decision problem is hard, then the optimization problem is hard too 373F19 - Nisarg Shah 25 Approximation Algorithms • There is a function 푃푟표푓푖푡 we want to maximize or a function 퐶표푠푡 we want to minimize • Given input instance 퐼… ➢ Our algorithm returns a solution 퐴퐿퐺(퐼) ➢ An optimal solution maximizing 푃푟표푓푖푡 or minimizing 퐶표푠푡 is 푂푃푇(퐼) ➢ Then, the approximation ratio of 퐴퐿퐺 on instance 퐼 is 푃푟표푓푖푡 푂푃푇 퐼 퐶표푠푡 퐿퐺 퐼 or 푃푟표푓푖푡 퐿퐺 퐼 퐶표푠푡 푂푃푇 퐼 373F19 - Nisarg Shah 26 Approximation Algorithms • Approximation ratio of 퐴퐿퐺 on instance 퐼 is 푃푟표푓푖푡 푂푃푇 퐼 퐶표푠푡 퐿퐺 퐼 or 푃푟표푓푖푡 퐿퐺 퐼 퐶표푠푡 푂푃푇 퐼 ➢ Note: These are defined to be ≥ 1 in each case. o 2-approximation = half the optimal profit / twice the optimal cost • 퐴퐿퐺 has worst-case 푐-approximation if for each instance 퐼… 1 푃푟표푓푖푡 퐴퐿퐺 퐼 ≥ ⋅ 푃푟표푓푖푡 푂푃푇 퐼 표푟 푐 퐶표푠푡 퐴퐿퐺 퐼 ≤ 푐 ⋅ 퐶표푠푡 푂푃푇 퐼 373F19 - Nisarg Shah 27 Local Search • A heuristic paradigm for solving complex problems ➢ Sometimes it might provably return an optimal solution ➢ But even if not, it might give a good approximation • Idea: ➢ Start with some solution 푆 ➢ While there is a “better” solution 푆′ in the local neighborhood of 푆 ➢ Switch to 푆’ • Need to define what is “better” and what is a “local neighborhood” 373F19 - Nisarg Shah 28 Randomized Algorithms Input Deterministic Algorithm Output Input Randomized Algorithm Output Randomness 373F19 - Nisarg Shah 29 Randomized Algorithms • Running time ➢ Sometimes, we want the algorithm to always take a small amount of time o Regardless of both the input and the random coin flips ➢ Sometimes, we want the algorithm to take a small amount of time in expectation o Expectation