Solving Systems of Linear Equations
Total Page:16
File Type:pdf, Size:1020Kb
Solving Systems of Linear Equations ● Motivation ● The idea behind Googles pagerank ● An example from economics ● Gaussian elimination ● LU Decomposition ● Iterative Methods – The Jacobi method ● Summary Motivation ● Systems of linear equations of the form A ⃗x=b⃗ or in components a11 x1+a12 x2+...+a1n xn=b1 a11 a12 ... a1n b1 a x +a x +...+a x =b a a ... a b 21 1 22 2 2n n 2 A= 21 22 2n ,b= 2 ... ... ... ... ... ... a x +a x +...+a x =b (am1 am2 ... amn) (bm) m1 1 m2 2 mn n m ● Are found in many contexts. For instance we need them to find roots of systems of non-linear equations using the Newton method. Short Reminder (1) ● Matrix-vector multiplication a11 a12 a13 x1 a11 x1+a12 x2+a13 x3 A ⃗x= a21 a22 a23 x2 = a21 x1+a22 x2+a23 x3 [ a31 a32 a33 ][ x3 ] [ a31 x1+a32 x2+a33 x3 ] ● Matrix-matrix multiplication a11 a12 a13 b11 b12 b13 ... ... ... = ... a b ... a21 a22 a23 b21 b22 b23 ∑k ik kj [a31 a32 a33 ][b31 b32 b33 ] [... ... ...] Short Reminder (2) ● Graphical illustration of 2d systems ● Every equation represents a line non-singular singular ● The following situations are possible – Lines cross at exactly one point -> unique solution (non- singular) – Lines are parallel (singular) and don't cross -> no solution – Lines overlap completely -> infinitely many solutions ● In higher d the same situations are possible Short Reminder (3) ● Focus on square matrices ● Some criteria that an n*n matrix is non-singular: ● The inverse of A (denoted A-1) exists ● det (A) !=0 ● rank (A) = n ● For any vector z!=0 it holds Az!=0 Example: How Google Search Works (sort of) ● In a particular group of people, who is the most popular? ● Say there are three people: Jane, Charlie, Fred ● We ask them to list their friends ● J lists C, C lists J,F, F lists J ● Could write this as a matrix J C F J 0 1 1 C 1 0 0 F 0 1 0 #total 1 2 1 Example (2) ● Some people list more than others as friends. To compensate we normalise each column by the sum J C F J 0 1/2 1 “linking matrix” C 1 0 0 =L F (0 1/2 0) ● Now want to find a vector p=(j,c,f) which assigns a popularity to each person ● Idea: a person's popularity is the weighted sum of the popularities of the people who mention it → p=Lp Example (3) 1 1/2 1/2 1/3 1/3 1/3 A node (person) is the more popular the ● more people list it as friends ● less friends these people list ● more popular these friends are (i.e. Having one very popular good friend can make you more popular than having 10 not so popular friends) Example (4) ● This defines a system of linear equations! J C F J 0 1/2 1 C 1 0 0 =L F (0 1/2 0) ● E.g.: j=1/2 c+ f, c=j, f=1/2 c or Lp= p ● In this case this is easy to solve ● System is under-determined: (e.g.) set j=1 as a scale, -> c=1, f=1/2, i.e. ● Charlie and Jane are most popular, Fred is less so. Remark ● We could use this to assign a popularity to web- pages ● The friend lists from before would then just refer to the lists of links in web-pages, i.e. How many web- pages link to a given page (normalized)? ● The popularity p could be interpreted as page rank (or we might wish to add contextual information (c) for search queries and not base them purely on popularity … e.g. p=Lp + c) ● We might wonder if the system Lp=p always has positive solutions p ● -> This is ensured by the Frobenius-Perron theorem Example: Setting Prices in an Economy ● This is a problem that goes back to Wassily Leontief (who got the Nobel prize for economics in 1973) ● Suppose in a far away land of Eigenbazistan, in a small country town called Matrixville, there lived a Farmer, a Tailor, a Carpenter, a Coal Miner and Slacker Bob. The Farmer produced food; the Tailor, clothes; the Carpenter, housing; the Coal Miner supplied energy; and Slacker Bob made High Quality 100% Proof Moonshine, half of which he drank himself. ● Let's assume that: ● No outside supply or demand ● Everything produced is consumed. ● Need to specify what fraction of each good is consumed by each person ● Problem: What are the incomes of each person? Consumption Food Clothes Housing Energy High Quality 100 Proof Moonshine Farmer 0.25 0.15 0.25 0.18 0.20 Tailor 0.15 0.28 0.18 0.17 0.05 Carpenter 0.22 0.19 0.22 0.22 0.10 Coal Miner 0.20 0.15 0.20 0.28 0.15 Slacker Bob 0.18 0.23 0.15 0.15 0.50 ● I.e. The farmer eats 25% of all food, 15% of all clothes, 25% of all housing, 18% of all energy ● Let's say price levels are p ,p ,p ,p ,p F T CA CO S ● Since this is a closed economy, we require: pF=0.25 pF+0.15 pT +0.25 pCA+0.18 pCO+0.20pS Again ... a System of Linear Equations pF=0.25 pF+0.15 pT +0.25 pCA+0.18 pCO+0.20pS pT =0.15 pF+0.28 pT +0.18 pCA +0.17 pCO +0.05pS pCA=0.22 pF+0.19 pT +0.22 pCA +0.22 pCO +0.10pS pCO=0.20 pF +0.15 pT +0.20 pCA +0.28 pCO +0.15pS pS=0.18 pF +0.23 pT +0.15 pCA +0.15 pCO +0.50pS ● Again this is a system of the type Ap=p (i.e. Under-determined, can set one price arbitrarily) ● Not so easy to solve by hand ... computer methods would be useful! Gaussian Elimination ● One idea to “mechanise” solving such systems is Gaussian elimination ● Consider the system: 2x+ y−z=8 −3x− y+2z=−11 −2x+ y+2z=−3 ● Observations: ● Can multiply equations by constants, swap equations, or add or subtract equations without changing solution ● Apply such operations to transform system into convenient form? ● What is a convenient form? -> triangular Gaussian Elimination (2) ● Triangular form: 2x+ y−z=8 (1) 1/3 y+1/3 z=2/3 (2) −z=1 (3) ● If our system had the above form we could just solve it by successive substitution, i.e. (3) z=−1 (substitute into) (2) 1/3 y−1/3=2/3→ y=3 (substitute into) (3) 2 x+3−(−1)=8→ x=2 How to Transform into Triangular Form? ● Start with: 2x+ y−z=8 (1) −3x− y+2z=−11 (2) −2x+ y+2z=−3 (3) ● Pivot (1). Multiply (2) by 2/3 and add to (1) 2x+ y−z=8 (1) y z + =2/3 (2') 3 3 −2x+ y+2z=−3 (3) ● Multiply (3) by 1 and add to (1) 2x+ y−z=8 (1) y z + =1 (2') 3 3 2y+z=5 (3') How to Transform into Triangular Form? (2) 2x+ y−z=8 (1) y z + =2/3 (2') 3 3 2y+z=5 (3') ● Now multiply (2') by -6 and add to (3') 2x+ y−z=8 (1) y z + =2/3 (2') 3 3 −z=1 (3'') ● Matrix is now in triangular form. Can diagonalise it as follows: Diagonalising ... ● Multiply (2') by 3 and add to (3''), (3'') by -1 add to (1): 2 x+ y =7 (1) y =3 (2'') −z=1 (3'') ● And finally: Multiply (2'') by -1 and add to (1). Finally, scale coefficients: 2 x =4 (1') y =3 (2'') z=1 (3'') Gaussian Elimination ● Usually, this is done using only coefficient schemes the “augmented” matrix ● Can be used to diagonalise matrices, just append identity matrix and perform all operations on it as well 2 1 −1 1 0 0 −3 −1 2 0 1 0 [−2 1 2 0 0 1] Coefficient matrix Identity matrix Example ... 2 1 −1 1 0 0 −3 −1 2 0 1 0 [−2 1 2 0 0 1] 2 1 −1 1 0 0 0 1/2 1/2 3/2 1 0 3/2*(1)+(2) [0 2 1 1 0 1] (1)+(2) 2 1 −1 1 0 0 0 1/2 1/2 3/2 1 0 [0 0 −1 −5 −4 1] -4*(2)+(3) Example (2) 2 1 −1 1 0 0 0 1/2 1/2 3/2 1 0 [0 0 −1 −5 −4 1] 2 1 0 6 4 −1 (1)-(3) 0 1 0 −2 −2 1 2*(2)+(3) [0 0 −1 −5 −4 1 ] 2 0 0 8 6 −2 (1)-(2) 0 1 0 −2 −2 1 -1 [0 0 −1 −5 −4 1 ] This is A 1 0 0 4 3 −1 1/2*(1) 0 1 0 −2 −2 1 1*(2) [0 0 1 5 4 −1] -1*(3) Example (3) ● OK, we found A-1, how to check if this is correct? ● Suppose we now have A-1 … how is this useful to find x? Some Remarks ● Pivot may be zero or (“inconvenient” for numerical stability) -> reorder rows ● Possibly divide by very small numbers (if pivot is small), better reorder to use largest possible pivot ● Computational complexity is of order O(n3) (roughly: n-1 rows need to be dealt with, this process involves order n operations and needs to be repeated less than n times) Pseudocode for k = 1 ..