Linear Separability
George M. Georgiou Linear Separability in Non-Euclidean Outline Geometry The problem
Separating hyperplane
Euclidean George M. Georgiou, Ph.D. Geometry
Non-Euclidean Geometry Computer Science Department California State University, San Bernardino Three solutions
Conclusion March 3, 2006
1 : 29 The problem
Linear Separability Find a geodesic that separates two given sets of points on George M. Georgiou the Poincare´ disk non-Euclidean geometry model.
Outline
The problem Linear Separability Geometrically
Separating hyperplane
Euclidean Geometry
Non-Euclidean Geometry
Three solutions
Conclusion
2 : 29 The problem
Linear Separability Find a geodesic that separates two given sets of points on George M. Georgiou the Poincare´ disk non-Euclidean geometry model.
Outline
The problem Linear Separability Geometrically
Separating hyperplane
Euclidean Geometry
Non-Euclidean Geometry
Three solutions
Conclusion
2 : 29 Linear Separability in Machine Learning
Linear Separability
George M. Georgiou Machine Learning is branch of AI, which includes areas Outline such as pattern recognition and artificial neural networks. The problem n Linear Separability Each point X = (x1, x2,..., xn) ∈ (vector or object) Geometrically R
Separating belongs to two class C0 and C1 with labels -1 and 1, hyperplane respectively. Euclidean Geometry The two classes are linearly separable if there exists n+1 Non-Euclidean W = (w0, w1, w2,..., wn) ∈ such that Geometry R
Three solutions w0 + w1x1 + w2x2 + ... + wnxn 0, if X ∈ C0 (1) Conclusion > w0 + w1x1 + w2x2 + ... + wnxn< 0, if X ∈ C1 (2)
3 : 29 Geometrically
Linear Separability
George M. Georgiou
Outline
The problem Linear Separability Geometrically
Separating hyperplane
Euclidean Geometry
Non-Euclidean Geometry
Three solutions
Conclusion
4 : 29 “Linearizing” the circle
Linear Separability
George M. Georgiou
Outline
The problem Linear Separability Geometrically
Separating hyperplane
Euclidean Geometry
Non-Euclidean Geometry (x, y) −→ (x, y, x2 + y 2) (3) Three solutions Conclusion Since any circle can be written as
2 2 w0 + w1x + w2y + w3(x + y ) = 0 (4)
5 : 29 Separating hyperplane
Linear Separability
George M. Georgiou
Outline
The problem Linear Separability Geometrically w0 + w1x1 + w2x2 + ... + wnxn= 0 Separating n = 2, 2-D: line hyperplane
Euclidean n = 3, 3-D: plane Geometry n > 3, higher dimensions: hyperplane Non-Euclidean Geometry
Three solutions
Conclusion
6 : 29 Finding the separating hyperplane
Linear Separability
George M. Georgiou
Outline
The problem A number of ways Separating hyperplane Use the perceptron algorithm Perceptron Linear inequalities Solve a system of linear inequalities Euclidean Geometry Use Support Vector Machine (SVM) method. It maximizes Non-Euclidean Geometry separation between the two classes.
Three solutions
Conclusion
7 : 29 Finding the separating hyperplane
Linear Separability
George M. Georgiou
Outline
The problem A number of ways Separating hyperplane Use the perceptron algorithm Perceptron Linear inequalities Solve a system of linear inequalities Euclidean Geometry Use Support Vector Machine (SVM) method. It maximizes Non-Euclidean Geometry separation between the two classes.
Three solutions
Conclusion
7 : 29 Finding the separating hyperplane
Linear Separability
George M. Georgiou
Outline
The problem A number of ways Separating hyperplane Use the perceptron algorithm Perceptron Linear inequalities Solve a system of linear inequalities Euclidean Geometry Use Support Vector Machine (SVM) method. It maximizes Non-Euclidean Geometry separation between the two classes.
Three solutions
Conclusion
7 : 29 The perceptron (Rosenblatt, 1958)
Linear w0 Separability 1 George M. x1 w1 Georgiou f(y) w2 x2 Outline w3 The problem x3 Separating w4 hyperplane Perceptron x4 Linear inequalities i=n Euclidean Geometry 1, if y 0 y = w x (6) f (y) = > (5) i i Non-Euclidean −1, if y < 0 i=0 Geometry X Three solutions Perceptron Algorithm Conclusion 1 Start with W = 0. 2 Present input vector X and compute error ε = d − f (y). 3 Update W using ∆W = α ε X 4 Pick another input vector X and goto to step 2.
8 : 29 Solving linear inequalities
Linear Separability
George M. Georgiou
Outline
The problem
Separating Can use linear programming, e.g. simplex method, hyperplane Karmakar, and variants. Perceptron Linear inequalities Polynomial complexity Euclidean Geometry In fact, when dimension is fixed, e.g. d=2,3 as in our case, Non-Euclidean Geometry complexity is linear in the number of input vectors.
Three solutions
Conclusion
9 : 29 Euclidean Geometry
Linear Separability George M. Axioms of Euclid (ca. 400 BC) Georgiou I. Two points determine a unique line. Outline
The problem II. A line is of infinite length. Separating III. A circle may be described with any center at any distance hyperplane from the center. Euclidean Geometry IV. All right angles are equal. Non-Euclidean Geometry V. If a straight line meet two other straight lines, so as to Three solutions make the two interior angles on one side of it together less Conclusion than two right angles, the other straight lines will meet if produced on that side on which the angles are less than two right angles.
(from H.S.M. Coxeter, Non-Euclidean Geometry)
10 : 29 Non-Euclidean Geometry
Linear Separability
George M. Georgiou Euclid’s Elements considered to be the most successful textbook of all times. Outline
The problem Nicolai Lobachevsky (1829) and Janos´ Bolyai (1832) first Separating to publish geometries without the Euclid’s axiom V. hyperplane
Euclidean Gauss feared the “howling of the Boetians.” Geometry
Non-Euclidean Geometry Types on Non-Euclidean Geometry Poincare´ Disk Lobachevskian (hyperbolic geometry) Geometry: from a Three solutions point not on a line, an infinite number of parallel lines can Conclusion be drawn. Riemannian (elliptic) geometry: no parallel lines can be drawn.
11 : 29 Poincare´ disk
Linear Separability
George M. Georgiou
Outline
The problem
Separating hyperplane
Euclidean Geometry
Non-Euclidean bM Geometry Poincare´ Disk bN Three solutions
Conclusion
12 : 29 Poincare´ disk
Linear Separability
George M. Georgiou
Outline
The problem
Separating hyperplane
Euclidean b P Geometry
Non-Euclidean bM Geometry Poincare´ Disk bN Three solutions
Conclusion
12 : 29 Distance function
Linear Separability
George M. Georgiou The distance between points z and z , taken as complex Outline 1 2
The problem numbers, is
Separating D(z1, z2) = | log(z1, z2; µ, λ)|, (7) hyperplane
Euclidean where z1 and z2 are points on the unit disk, and µ and λ are Geometry the intersections of the geodesic defined by z1 and z2 with the Non-Euclidean Geometry unit circle. The cross ratio (z1, z2; µ, λ), is defined as Poincare´ Disk Three solutions z − µ.z − µ (z , z ; µ, λ) = 1 2 (8) Conclusion 1 2 z1 − λ z2 − λ
13 : 29 Art by M.C. Esher (1878–1972) Limit Circle III
Linear Separability
George M. Georgiou
Outline
The problem
Separating hyperplane
Euclidean Geometry
Non-Euclidean Geometry Poincare´ Disk
Three solutions
Conclusion
14 : 29
The problem
Linear Separability Find a geodesic that separates two given sets of points on George M. Georgiou the Poincare´ disk non-Euclidean geometry model.
Outline
The problem
Separating hyperplane
Euclidean Geometry
Non-Euclidean Geometry Poincare´ Disk
Three solutions
Conclusion
16 : 29 The problem
Linear Separability Find a geodesic that separates two given sets of points on George M. Georgiou the Poincare´ disk non-Euclidean geometry model.
Outline
The problem
Separating hyperplane
Euclidean Geometry
Non-Euclidean Geometry Poincare´ Disk
Three solutions
Conclusion
16 : 29 Three solutions
Linear Separability
George M. Georgiou
Outline 1 Map points from Poincare´ disk to the Klein disk, find The problem separating line, which maps to a desired geodesic on the Separating Poincare´ disk. hyperplane 2 Euclidean Map points from Poincare´ disk to the Hyperboloid model, Geometry find separating plane, which maps to a desired geodesic Non-Euclidean Geometry on the Poincare´ disk. Three solutions 3 Map points from Poincare´ disk to circular paraboloid, find Klein disk Hyperboloid model separating plane, which maps to a desired geodesic on Circular Paraboloid
Conclusion the Poincare´ disk. (This is a novel map.)
17 : 29 Three solutions
Linear Separability
George M. Georgiou
Outline 1 Map points from Poincare´ disk to the Klein disk, find The problem separating line, which maps to a desired geodesic on the Separating Poincare´ disk. hyperplane 2 Euclidean Map points from Poincare´ disk to the Hyperboloid model, Geometry find separating plane, which maps to a desired geodesic Non-Euclidean Geometry on the Poincare´ disk. Three solutions 3 Map points from Poincare´ disk to circular paraboloid, find Klein disk Hyperboloid model separating plane, which maps to a desired geodesic on Circular Paraboloid
Conclusion the Poincare´ disk. (This is a novel map.)
17 : 29 Three solutions
Linear Separability
George M. Georgiou
Outline 1 Map points from Poincare´ disk to the Klein disk, find The problem separating line, which maps to a desired geodesic on the Separating Poincare´ disk. hyperplane 2 Euclidean Map points from Poincare´ disk to the Hyperboloid model, Geometry find separating plane, which maps to a desired geodesic Non-Euclidean Geometry on the Poincare´ disk. Three solutions 3 Map points from Poincare´ disk to circular paraboloid, find Klein disk Hyperboloid model separating plane, which maps to a desired geodesic on Circular Paraboloid
Conclusion the Poincare´ disk. (This is a novel map.)
17 : 29 The Klein disk
Linear Separability
George M. Georgiou
Outline
The problem
Separating Points, like those of the Poincare´ disk, are the usual hyperplane Euclidean points. Euclidean Geometry Lines are the familiar straight Euclidean lines, but of Non-Euclidean course are bounded but the unit circle. Geometry
Three solutions Angles, unlike those in the Poincare´ disk, are distorted. Klein disk Hyperboloid model Circular Paraboloid
Conclusion
18 : 29 The Klein disk
Linear Separability
George M. Georgiou
Outline
The problem
Separating Points, like those of the Poincare´ disk, are the usual hyperplane Euclidean points. Euclidean Geometry Lines are the familiar straight Euclidean lines, but of Non-Euclidean course are bounded but the unit circle. Geometry
Three solutions Angles, unlike those in the Poincare´ disk, are distorted. Klein disk Hyperboloid model Circular Paraboloid
Conclusion
18 : 29 The Klein disk
Linear Separability
George M. Georgiou
Outline
The problem
Separating Points, like those of the Poincare´ disk, are the usual hyperplane Euclidean points. Euclidean Geometry Lines are the familiar straight Euclidean lines, but of Non-Euclidean course are bounded but the unit circle. Geometry
Three solutions Angles, unlike those in the Poincare´ disk, are distorted. Klein disk Hyperboloid model Circular Paraboloid
Conclusion
18 : 29 The two geodesics
Linear Separability
George M. B Georgiou
Outline
The problem
Separating hyperplane
Euclidean Geometry
Non-Euclidean Geometry
Three solutions Klein disk Hyperboloid model Circular Paraboloid A Conclusion Blue: Poincare´ Red: Klein
19 : 29 From the Poincare´ to the Klein disk, and back
Linear (0,0,1) Separability
George M. Georgiou
Outline
The problem Separating P hyperplane K Euclidean Geometry
Non-Euclidean Geometry
Three solutions Klein disk Hyperboloid model Circular Paraboloid Conclusion 2x 2y F(x, y) = , (9) 1 + x2 + y 2 1 + x2 + y 2 ! −1 x y F (x, y) = p , p (10) 1 + 1 − x2 − y 2 1 + 1 − x2 − y 2 20 : 29 1st Solution
Linear Separability
George M. Georgiou
Outline The problem 1 Map all given points from the Poincare´ disk to the Klein Separating hyperplane disk Euclidean 2 Find a separating line Geometry 3 Non-Euclidean Map the two points of the intersection of the line and the Geometry unit circle to the Poincare´ disk Three solutions Klein disk 4 Find geodesic that corresponds to the two points Hyperboloid model Circular Paraboloid
Conclusion
21 : 29 The Hyperboloid model
Linear Separability
George M. p 2 2 Georgiou H = {(x, y, x + y + 1): x, y ∈ R} (11)
Outline
The problem
Separating hyperplane
Euclidean Geometry
Non-Euclidean Geometry
Three solutions Klein disk Hyperboloid model Circular Paraboloid
Conclusion
Geodesics are intersections of planes through the origin.
22 : 29 From Poincare´ to the Hyperboloid, and back
Linear Separability
George M. Georgiou
Outline The problem (x,y,z) Separating hyperplane Klein Disk
Euclidean Geometry
Non-Euclidean Poincar´eDisk Geometry P
Three solutions Klein disk Hyperboloid model Circular Paraboloid P = r + i s (12) Conclusion 2r 2s 1 + |P|2 F(P) = , , (13) 1 − |P|2 1 − |P|2 1 − |P|2 x + i y F −1(x, y, z) = (14) 1 + z 23 : 29 2nd Solution
Linear Separability
George M. Georgiou
Outline The problem 1 Map all given points from the Poincare´ disk to the Separating hyperplane hyperboloid Euclidean 2 Find a separating plane that passes through the origin Geometry 3 Non-Euclidean Map two points on the intersection of the plane and the Geometry hyperboloid to the Poincare´ disk Three solutions Klein disk 4 Find geodesic that corresponds to the two points Hyperboloid model Circular Paraboloid
Conclusion
24 : 29 The circular paraboloid I
Linear Separability 2 2 2 2 George M. Ω = {(x, y, x + y + 1): x, y ∈ R, x + y < 1} (15) Georgiou
Outline
The problem
Separating hyperplane
Euclidean Geometry
Non-Euclidean Geometry (x,y,z) Three solutions Klein disk Hyperboloid model Circular Paraboloid Conclusion (x,y) Fact: Circles on the x-y plane are projections of ellipses on f (x, y) = x2 + y 2 + 1. (D. Pedoe, Geometry, 1988) shows this for the surface (x, y, x2 + y 2).
25 : 29 The circular paraboloid II
Linear Separability
George M. Georgiou
Outline
The problem
Separating (x,y,z) hyperplane
Euclidean Geometry (x,y) Non-Euclidean Geometry It can be shown: the ellipse on f (x, y) = x2 + y 2 + 1 is the Three solutions Klein disk intersection of a plane that passes through the origin. Hyperboloid model Circular Paraboloid It can be shown: the corresponding circles on the x-y Conclusion plane are orthogonal to the unit circle. A geodesic on the Poincare´ disk maps to an arc of an ellipse on Ω that lies on a plane through the origin.
26 : 29 3rd Solution
Linear Separability
George M. Georgiou
Outline
The problem 1 Map all given points from the Poincare´ disk to Ω Separating hyperplane 2 Find a separating plane that passes through the origin Euclidean Geometry 3 Map two points on the intersection of the plane and Ω to Non-Euclidean the Poincare´ disk (just discard the z coordinate) Geometry Three solutions 4 Find geodesic that corresponds to the two points Klein disk Hyperboloid model Circular Paraboloid
Conclusion
27 : 29 Advantages of the 3rd solution
Linear Separability
George M. Georgiou
Outline
The problem
Separating hyperplane Mapping to and from is simple. Euclidean Geometry No division necessary Non-Euclidean Unlike 2nd method, no overflow possible in mapping Geometry
Three solutions Klein disk Hyperboloid model Circular Paraboloid
Conclusion
28 : 29 Conclusion
Linear Separability
George M. Georgiou
Outline
The problem
Separating hyperplane Three ways given two find a geodesic that separates two Euclidean classes of vectors on the Poincare´ disk. Geometry
Non-Euclidean A new way to visualize the Poincare´ disk is given. Geometry
Three solutions
Conclusion
29 : 29