CS4102 Algorithms Summer 2020 Warm Up How many arithmetic operations are required to multiply a 푛 × 푚 Matrix with a 푚 × 푝 Matrix? (don’t overthink this) 푝 푚 푚 푛 × 1 Warm Up How many arithmetic operations are required to multiply a 푛 × 푚 Matrix with a 푚 × 푝 Matrix? (don’t overthink this) 푝 푝 푚 푛 × 푚 = 푛 • 푚 multiplications and additions per element • 푛 ⋅ 푝 elements to compute • Total cost: 푚 ⋅ 푛 ⋅ 푝 2 Dynamic Programming • Requires Optimal Substructure – Solution to larger problem contains the (optimal) solutions to smaller ones • Idea: 1. Identify the recursive structure of the problem • What is the “last thing” done? 2. Save the solution to each subproblem in memory 3. Select a good order for solving subproblems • “Top Down”: Solve each recursively • “Bottom Up”: Iteratively solve smallest to largest 3 Matrix Chaining • Given a sequence of Matrices (푀1, … , 푀푛), what is the most efficient way to multiply them? 푐4 푐2 푐 1 푐3 푟 × 푟 푀 × 푀 푟1 푀1 × 2 푀2 3 3 푟4 4 4 Order Matters! 푐2 푐3 푐1 = 푟2 푐1 푐2 = 푟3 푟 × 푟 푟1 푀1 × 2 푀2 3 푀3 푐2 푟1 • 푀1 × 푀2 × 푀3 – uses 푐1 ⋅ 푟1 ⋅ 푐2 + c2 ⋅ 푟1 ⋅ 푐3 operations 5 Order Matters! 푐2 푐3 푐1 = 푟2 푐1 푐2 = 푟3 푟 × 푟 푟1 푀1 × 2 푀2 3 푀3 푐3 푟2 • 푀1 × (푀2 × 푀3) – uses c1 ⋅ r1 ⋅ 푐3 + (c2 ⋅ 푟2 ⋅ 푐3) operations 6 Order Matters! 푐1 = 푟2 푐2 = 푟3 • 푀1 × 푀2 × 푀3 – uses 푐1 ⋅ 푟1 ⋅ 푐2 + c2 ⋅ 푟1 ⋅ 푐3 operations 푀1 = 7 × 10 – 10 ⋅ 7 ⋅ 20 + 20 ⋅ 7 ⋅ 8 = 2520 푀2 = 10 × 20 • 푀1 × (푀2 × 푀3) 푀3 = 20 × 8 – uses 푐1 ⋅ 푟1 ⋅ 푐3 + (c2 ⋅ 푟2 ⋅ 푐3) operations 푐1 = 10 푐2 = 20 – 10 ⋅ 7 ⋅ 8 + 20 ⋅ 10 ⋅ 8 = 2160 푐3 = 8 푟1 = 7 푟2 = 10 푟3 = 20 7 Dynamic Programming • Requires Optimal Substructure – Solution to larger problem contains the solutions to smaller ones • Idea: 1. Identify the recursive structure of the problem • What is the “last thing” done? 2. Save the solution to each subproblem in memory 3. Select a good order for solving subproblems • “Top Down”: Solve each recursively • “Bottom Up”: Iteratively solve smallest to largest 8 1. Identify the Recursive Structure of the Problem 퐵푒푠푡 1, 푛 = cheapest way to multiply together 푀1 through 푀푛 푐4 푐2 푐 1 푐3 푟 푀 × 푟 푀 × 푀 푟1 푀1 × 2 2 3 3 푟4 4 9 1. Identify the Recursive Structure of the Problem 퐵푒푠푡 1, 푛 = cheapest way to multiply together 푀1 through 푀푛 퐵푒푠푡 2,4 + 푟1푟2푐4 퐵푒푠푡 1,4 = min 푐4 푟2 푐4 푐2 푐 1 푐3 푟 푀 × 푟 푀 × 푀 푟1 푀1 × 2 2 3 3 푟4 4 10 1. Identify the Recursive Structure of the Problem 퐵푒푠푡 1, 푛 = cheapest way to multiply together 푀1 through 푀푛 퐵푒푠푡 2,4 + 푟1푟2푐4 퐵푒푠푡 1,4 = min 퐵푒푠푡 1,2 + 퐵푒푠푡 3, 4 + 푟1푟3푐4 푐 푐2 4 푟3 푟1 푐4 푐2 푐 1 푐3 푟 푀 × 푟 푀 × 푀 푟1 푀1 × 2 2 3 3 푟4 4 11 1. Identify the Recursive Structure of the Problem 퐵푒푠푡 1, 푛 = cheapest way to multiply together 푀1 through 푀푛 퐵푒푠푡 2,4 + 푟1푟2푐4 퐵푒푠푡 1,4 = min 퐵푒푠푡 1,2 + 퐵푒푠푡 3, 4 + 푟1푟3푐4 퐵푒푠푡 1,3 + 푟1푟4푐4 푐3 푟1 푐4 푐2 푐 1 푐3 푟 푀 × 푟 푀 × 푀 푟1 푀1 × 2 2 3 3 푟4 4 12 1. Identify the Recursive Structure of the Problem • In general: 퐵푒푠푡 푖, 푗 = cheapest way to multiply together 푀푖 through 푀푗 푗−1 퐵푒푠푡 푖, 푗 = min 퐵푒푠푡 푖, 푘 + 퐵푒푠푡 푘 + 1, 푗 + 푟푖푟푘+1푐푗 푘=푖 퐵푒푠푡 푖, 푖 = 0 퐵푒푠푡 2, 푛 + 푟1푟2푐푛 퐵푒푠푡 1,2 + 퐵푒푠푡 3, 푛 + 푟1푟3푐푛 퐵푒푠푡 1,3 + 퐵푒푠푡 4, 푛 + 푟1푟4푐푛 퐵푒푠푡 1, 푛 = min 퐵푒푠푡 1,4 + 퐵푒푠푡 5, 푛 + 푟1푟5푐푛 … 퐵푒푠푡 1, 푛 − 1 + 푟1푟푛푐푛 13 Dynamic Programming • Requires Optimal Substructure – Solution to larger problem contains the solutions to smaller ones • Idea: 1. Identify the recursive structure of the problem • What is the “last thing” done? 2. Save the solution to each subproblem in memory 3. Select a good order for solving subproblems • “Top Down”: Solve each recursively • “Bottom Up”: Iteratively solve smallest to largest 14 2. Save Subsolutions in Memory • In general: 퐵푒푠푡 푖, 푗 = cheapest way to multiply together 푀푖 through 푀푗 푗−1 퐵푒푠푡 푖, 푗 = min 퐵푒푠푡 푖, 푘 + 퐵푒푠푡 푘 + 1, 푗 + 푟푖푟푘+1푐푗 푘=푖 퐵푒푠푡 푖, 푖 = 0 Read from M[i,j] if present Save to M[i,j] 퐵푒푠푡 2, 푛 + 푟1푟2푐푛 퐵푒푠푡 1,2 + 퐵푒푠푡 3, 푛 + 푟1푟3푐푛 퐵푒푠푡 1,3 + 퐵푒푠푡 4, 푛 + 푟1푟4푐푛 퐵푒푠푡 1, 푛 = min 퐵푒푠푡 1,4 + 퐵푒푠푡 5, 푛 + 푟1푟5푐푛 … 퐵푒푠푡 1, 푛 − 1 + 푟1푟푛푐푛 15 Dynamic Programming • Requires Optimal Substructure – Solution to larger problem contains the solutions to smaller ones • Idea: 1. Identify the recursive structure of the problem • What is the “last thing” done? 2. Save the solution to each subproblem in memory 3. Select a good order for solving subproblems • “Top Down”: Solve each recursively • “Bottom Up”: Iteratively solve smallest to largest 16 3. Select a good order for solving subproblems • In general: 퐵푒푠푡 푖, 푗 = cheapest way to multiply together 푀푖 through 푀푗 푗−1 퐵푒푠푡 푖, 푗 = min 퐵푒푠푡 푖, 푘 + 퐵푒푠푡 푘 + 1, 푗 + 푟푖푟푘+1푐푗 푘=푖 퐵푒푠푡 푖, 푖 = 0 Read from M[n] if present Save to M[n] 퐵푒푠푡 2, 푛 + 푟1푟2푐푛 퐵푒푠푡 1,2 + 퐵푒푠푡 3, 푛 + 푟1푟3푐푛 퐵푒푠푡 1,3 + 퐵푒푠푡 4, 푛 + 푟1푟4푐푛 퐵푒푠푡 1, 푛 = min 퐵푒푠푡 1,4 + 퐵푒푠푡 5, 푛 + 푟1푟5푐푛 … 퐵푒푠푡 1, 푛 − 1 + 푟1푟푛푐푛 17 3. Select a good order for solving subproblems 35 15 5 10 20 25 푀4 푀3 푀5 30 푀1 × 35 × 15 × 5 × 10 × 20 푀6 푀2 푗−1 퐵푒푠푡 푖, 푗 = min 퐵푒푠푡 푖, 푘 + 퐵푒푠푡 푘 + 1, 푗 + 푟푖푟푘+1푐푗 푘=푖 푗 = 1 2 3 4 5 6 퐵푒푠푡 푖, 푖 = 0 0 1 0 2 0 3 0 4 0 5 0 6 18 3. Select a good order for solving subproblems 35 15 5 10 20 25 푀4 푀3 푀5 30 푀1 × 35 × 15 × 5 × 10 × 20 푀6 푀2 푗−1 퐵푒푠푡 푖, 푗 = min 퐵푒푠푡 푖, 푘 + 퐵푒푠푡 푘 + 1, 푗 + 푟푖푟푘+1푐푗 푘=푖 푗 = 1 2 3 4 5 6 퐵푒푠푡 푖, 푖 = 0 0 15750 1 0 2 0 3 0 4 퐵푒푠푡 1,1 + 퐵푒푠푡 2, 2 + 푟 푟 푐 0 5 퐵푒푠푡 1,2 = min 1 2 2 0 6 19 3. Select a good order for solving subproblems 35 15 5 10 20 25 푀4 푀3 푀5 30 푀1 × 35 × 15 × 5 × 10 × 20 푀6 푀2 푗−1 퐵푒푠푡 푖, 푗 = min 퐵푒푠푡 푖, 푘 + 퐵푒푠푡 푘 + 1, 푗 + 푟푖푟푘+1푐푗 푘=푖 푗 = 1 2 3 4 5 6 퐵푒푠푡 푖, 푖 = 0 0 15750 1 0 2625 2 0 3 0 4 퐵푒푠푡 2,2 + 퐵푒푠푡 3, 3 + 푟 푟 푐 0 5 퐵푒푠푡 2,3 = min 2 3 3 0 6 20 3. Select a good order for solving subproblems 35 15 5 10 20 25 푀4 푀3 푀5 30 푀1 × 35 × 15 × 5 × 10 × 20 푀6 푀2 푗−1 퐵푒푠푡 푖, 푗 = min 퐵푒푠푡 푖, 푘 + 퐵푒푠푡 푘 + 1, 푗 + 푟푖푟푘+1푐푗 푘=푖 푗 = 1 2 3 4 5 6 퐵푒푠푡 푖, 푖 = 0 0 15750 1 0 2625 2 0 750 3 0 1000 4 0 5000 5 0 6 21 3. Select a good order for solving subproblems 35 15 5 10 20 25 푀4 푀3 푀5 30 푀1 × 35 × 15 × 5 × 10 × 20 푀6 푀2 푗−1 퐵푒푠푡 푖, 푗 = min 퐵푒푠푡 푖, 푘 + 퐵푒푠푡 푘 + 1, 푗 + 푟푖푟푘+1푐푗 푘=푖 푗 = 1 2 3 4 5 6 퐵푒푠푡 푖, 푖 = 0 0 15750 7875 1 0 2625 2 푟1푟2푐3 = 30 ⋅ 35 ⋅ 5 = 5250 푟1푟3푐3 = 30 ⋅ 15 ⋅ 5 = 2250 0 750 3 0 1000 4 0 2625 0 5000 5 퐵푒푠푡 1,3 = min 퐵푒푠푡 1,1 + 퐵푒푠푡 2, 3 + 푟1푟2푐3 퐵푒푠푡 1,2 + 퐵푒푠푡 3, 3 + 푟1푟3푐3 0 6 15750 0 22 3. Select a good order for solving subproblems 35 15 5 10 20 25 푀4 푀3 푀5 30 푀1 × 35 × 15 × 5 × 10 × 20 푀6 푀2 푗−1 퐵푒푠푡 푖, 푗 = min 퐵푒푠푡 푖, 푘 + 퐵푒푠푡 푘 + 1, 푗 + 푟푖푟푘+1푐푗 푘=푖 푗 = 1 2 3 4 5 6 퐵푒푠푡 푖, 푖 = 0 0 15750 7875 1 0 2625 2 To find 퐵푒푠푡(푖, 푗): Need all preceding 0 750 3 terms of row 푖 and column 푗 0 1000 4 Conclusion: solve in order of diagonal 0 5000 5 0 6 23 1 2 3 4 5 6 0 15750 7875 1 0 2625 2 0 750 3 0 1000 4 1 2 3 4 5 6 0 5000 5 0 15750 7875 1 0 6 0 2625 2 0 750 3 0 1000 4 0 5000 5 0 6 Matrix Chaining 35 15 5 10 20 25 푀4 푀3 푀5 30 푀1 × 35 × 15 × 5 × 10 × 20 푀6 푀2 푗−1 퐵푒푠푡 푖, 푗 = min 퐵푒푠푡 푖, 푘 + 퐵푒푠푡 푘 + 1, 푗 + 푟푖푟푘+1푐푗 푘=푖 푗 = 1 2 3 4 5 6 퐵푒푠푡 푖, 푖 = 0 0 15750 7875 9375 11875 15125 1 0 2625 4375 7125 10500 2 0 750 2500 5375 3 퐵푒푠푡 1,1 + 퐵푒푠푡 2, 6 + 푟1푟2푐6 0 1000 3500 4 퐵푒푠푡 1,2 + 퐵푒푠푡 3, 6 + 푟1푟3푐6 퐵푒푠푡 1,6 = min 퐵푒푠푡 1,3 + 퐵푒푠푡 4, 6 + 푟1푟4푐6 0 5000 5 퐵푒푠푡 1,4 + 퐵푒푠푡 5, 6 + 푟1푟5푐6 0 6 퐵푒푠푡 1,5 + 퐵푒푠푡 6, 6 + 푟1푟6푐6 25 Run Time 1.