DOCSLIB.ORG

Linear Classifiers, a Perceptron Family

Total Page：16

File Type：pdf, Size：1020Kb

Download full-text PDF Read full-text
Linear Classifiers, a Perceptron Family Linear classifiers, a perceptron family Václav Hlaváč Czech Technical University in Prague Czech Institute of Informatics, Robotics and Cybernetics 160 00 Prague 6, Jugoslávských partyzánů 1580/3, Czech Republic http://people.ciirc.cvut.cz/hlavac, [email protected] also Center for Machine Perception, http://cmp.felk.cvut.cz Courtesy: M.I. Schlesinger, V. Franc. Outline of the talk: A classifier, dichotomy, a multi-class classifier. Perceptron and its learning. A linear discriminant function. ε-solution. Learning a linear classifier. Learning for infinite training sets. A classifier 2/23 Analyzed object is represented by X – a space of observations, a vector space of dimension n. Y – a set of hidden states. The aim of the classification is to determine a relation between X and Y , i.e. to find a discriminant function f : X → Y . Classifier q : X → J maps observations Xn → set of class indices J, J = 1,..., |Y |. Mutual exclusion of classes is required X = X1 ∪ X2 ∪ ... ∪ X|Y |, Xi ∩ Xj = ∅, i 6= j, i, j = 1 ... |Y |. Classifier, an illustration 3/23 A classifier partitions the observation space X into class-labelled regions Xi, i = 1,..., |Y |. Classification determines, to which region Xi corresponding to a hidden state yi an observed feature vector x belongs. Borders between regions are called decision boundaries. X 1 X X1 X 4 X X 3 X2 3 1 X2 X3 X2 X4 X4 - background Several possible arrangements of decision boundaries corresponding to hidden states yi (classes in a more specific case). A multi-class decision strategy 4/23 Discriminant functions fi(x) should have the (ideal) property: fi(x) > fj(x) for x ∈ class i , i 6= j. f1 (x) x f (x) y 2 max f||Y (x) Strategy: j = argmax fj(x) j However, it is uneasy to find such a discriminant function. Most ‘good classifiers’ are dichotomic (as perceptron, SVM). The usual solution: One-against-All classifier, One-against-One classifier. Linear discriminant function 5/23 fj(x) = hwj, xi + bj, where h i denotes a scalar product. A strategy j = argmax fj(x) divides X into |Y | convex regions. j y=6 y=5 y=1 y=2 y=4 y=3 The classes separation corresponds to the Voronoi diagram (a term from computational geometry). The aim is to obtain a partition filling the space completely and reflecting the proximity information. Voronoi diagram is a partition of a space (plane in a special case) into regions close to each of given objects (seeds). The region corresponding to a particular seed consists of all points closer to that seed than to other seeds. Dichotomy, two classes only 6/23 |Y | = 2, i.e. two hidden states (typically also classes) y = 1 , if hw, xi + b ≥ 0 , q(x) = y = 2 , if hw, xi + b < 0 . x 1 1 b threshold activation x1 w1 function x w 2 2 y x w 3 3 S x2 xn wn weights Perceptron by F. Rosenblatt 1958 Learning linear classifiers 7/23 The aim of learning is to estimate classifier q(x) parameters wi, bi for ∀i. The learning algorithms differ by The character of training set 1. Finite set consisting of individual observations and hidden states, i.e., {(x1, y1) ... (xL, yL)}. 2. Infinite sets described e.g. by a mixture of Gaussian distributions. Learning task formulations. Learning tasks formulations for finite training sets 8/23 Empirical risk minimization: dummy dummy dummy dummy dummy L 1 P True risk is approximated by Remp(q(x, Θ)) = L W (q(xi, Θ), yi), where W is a i=1 penalty. ∗ Learning is based on the empirical minimization principle Θ = argmin Remp(q(x, Θ)). Θ Examples of learning algorithms: Perceptron, Back-propagation. Structural risk minimization: dummy dummy dummy dummy dummy True risk is approximated by a guaranteed risk (a regularizer securing upper bound of the risk is added to the empirical risk, Vapnik-Chervonenkis theory of learning). Example: Support Vector Machine (SVM). Perceptron, the history 9/23 The perceptron (F. Rosenblatt, 1958) was intended to be a machine (even it was first implemented as a program on IBM 704). This machine was designed for image recognition: it had an array of 400 photocells, randomly connected to the “neurons”. Weights were encoded in potentiometers, and weight updates during learning were performed by electric motors Although the perceptron initially seemed promising, its limitations appeared. In 1969 a famous book entitled Perceptrons by Marvin Minsky and Seymour Papert showed that it was impossible for these classes of network to learn an XOR function. It is often believed (incorrectly) that they also conjectured that a similar result would hold for a multi-layer perceptron network. The often-miscited Minsky/Papert text caused a significant decline in interest and funding of neural network research. Courtesy: Wikipedia (A single-layer) perceptron learning; Task formulation 10/23 Input: T = {(x1, y1) ... (xL, yL)}, yi ∈ {1, 2}, i = 1,...,L, w, x + b = 0 dim(xi) = n. X n Output: a weight vector w, offset b for ∀j ∈ {1,...,L} satis- fying: hw, xji + b ≥ 0 for y = 1, hw, xji + b < 0 for y = 2. The task can be formally transcribed to a single inequality 0 0 hw , xji ≥ 0 by embedding it into n + 1 dimensional space, X n+1 w’, x’ = 0 where w0 = [w b], [x 1] for y = 1 , x0 = −[x 1] for y = 2 . We drop the primes and go back to w, x notation. Perceptron learning: the algorithm 1957 11/23 Input: T = {(x1, y1) ... (xL, yL)}. w Output: a weight vector w. t wt+1 The Perceptron algorithm 1. w1 = 0. w,x=0t 2. A wrongly classified observation xj is sought, i.e., hwt, xji < 0, j ∈ {1,...,L}. 0 xt 3. If there is no misclassified observation then Perceptron update rule the algorithm terminates otherwise wt+1 = wt + xj. 4. Goto2. The perceptron algorithm was invented at the F. Rosenblatt: The perceptron: a probabilistic model for information storage and Cornell Aeronautical Laboratory by Frank Ro- organization in the brain,”Psychological review 65.6 (1958): 386 senblatt (∗1928, †1971). Novikoff theorem, 1962 12/23 Proves that the Perceptron algorithm converges in a finite number steps if the linearly separable solution exists. convex hull Let X denotes a convex hull of points (set of observations) X. Let D = max |xi|, m = min |xi| > 0. i x∈X Novikoff theorem: If the data is linearly separable then there exists a ∗ D2 number t ≤ , such that the vector w ∗ satisfies m2 t D hwt∗, xji > 0, ∀j ∈ {1,...,L} . m origin What if the data is not separable? How to terminate the perceptron learning? Idea of the Novikoff theorem proof 13/23 2 Let express bounds for |wt| : 2 Upper bound: |wt | 2 2 2 2 |wt+1| = |wt + xt| = |wt| + 2 hxt, wti +|xt| | ≤{z0 } 2 2 2 2 ≤ |wt| + |xt| ≤ |wt| + D . 2 2 2 2 2 2 2 |w0| = 0, |w1| ≤ D , |w2| ≤ 2D ,..., |wt+1| ≤ t D , ... 2 2 2 Lower bound: is given analogically |wt+1| >t m . t 2 2 2 D2 Solution: t m ≤ t D ⇒ t ≤ m2 . An alternative training algorithm Kozinec (1973) 14/23 Input: T = {(x1, y1) ... (xL, yL)}. wt Output: a weight vector w∗. w t+1 w,x=0 1. w = x , i.e., any observation. t 1 j b 2. A wrongly classified observation xt is sought, i.e., j x hwt, x i < 0, j ∈ J. t 0 3. If there is no wrongly classified observation then the algorithm finishes otherwise Kozinec wt+1 = (1 − k) · wt + xt · k, k ∈ R, where k = argmin |(1 − k) · wt + xt · k|. k 4. Goto2. Perceptron learning as an optimization problem (1) 15/23 Perceptron algorithm, the batch version, handling non-separability, another perspective: Input: T = {(x1, y1) ... (xL, yL)}, yi ∈ {1, 2}, i = 1,...,L, dim(xi) = n. Output: a weight vector w minimizing J(w) = |{x ∈ X : hwt, xi < 0}| or, equivalently X J(w) = 1 . x∈X : hwt,xi<0 Q: Is the most common optimization method, the gradient descent wt = w − η∇J(w), applicable? A: Unfortunately not. The gradient of J(w) is either 0 or undefined. Perceptron learning as an Optimization problem (2) 16/23 Let us redefine the cost function: X Jp(w) = hw, xi . x∈X : hw,xi<0 ∂J X ∇J (w) = = x . p ∂w x∈X : hw,xi<0 The Perceptron algorithm performs gradient descent optimization of Jp(w). Learning by the empirical risk minimization is just an instance of an optimization problem. Either gradient minimization (backpropagation in neural networks) or convex (quadratic) minimization (called convex programming in mathematical literature) is used. Perceptron algorithm 17/23 Classifier learning, a non-separable case, a batch version Input: T = {(x1, y1) ... (xL, yL)}, yi ∈ {1, 2}, i = 1,...,L, dim(xi) = n. Output: a weight vector w∗. ∗ 1. w1 = 0, E = |T | = L, w = 0 . − 2. Find all misclassified observations X = {x ∈ X : hwt, xi < 0}. − − ∗ 3. If |X | < E then E = |X |; w = wt, the time of the last update tlu = t. ∗ P 4. If the termination condition tc(w , t, tlu) then terminate else wt+1 = wt + ηt x. x∈X− 5. Goto2. The algorithm converges with the probability 1 to the optimal solution. The convergence rate is not known. The termination condition tc(.) is a complex function of the quality of the best solution, time since the last update t − tlu and requirements on the solution. The optimal separating plane and the closest point to the convex hull 18/23 The problem of the optimal separation by a hyperplane ∗ w w = argmax min , xj (1) w j |w| can be converted to a seek for the closest point to a convex hull (denoted by the overline) of the observations x in the training multi-set.
Load more

Recommended publications
  • Warm Start for Parameter Selection of Linear Classifiers
    Warm Start for Parameter Selection of Linear Classifiers Bo-Yu Chu Chia-Hua Ho Cheng-Hao Tsai Dept. of Computer Science Dept. of Computer Science Dept. of Computer Science National Taiwan Univ., Taiwan National Taiwan Univ., Taiwan National Taiwan Univ., Taiwan [email protected] [email protected] [email protected] Chieh-Yen Lin Chih-Jen Lin Dept. of Computer Science Dept. of Computer Science National Taiwan Univ., Taiwan National Taiwan Univ., Taiwan [email protected] [email protected] ABSTRACT we may need to solve many optimization problems. Sec- In linear classification, a regularization term effectively reme- ondly, if we do not know the reasonable range of the pa- dies the overfitting problem, but selecting a good regulariza- rameters, we may need a long time to solve optimization tion parameter is usually time consuming. We consider cross problems under extreme parameter values. validation for the selection process, so several optimization In this paper, we consider using warm start to efficiently problems under different parameters must be solved. Our solve a sequence of optimization problems with different reg- aim is to devise effective warm-start strategies to efficiently ularization parameters. Warm start is a technique to reduce solve this sequence of optimization problems. We detailedly the running time of iterative methods by using the solution investigate the relationship between optimal solutions of lo- of a slightly different optimization problem as an initial point gistic regression/linear SVM and regularization parameters. for the current problem. If the initial point is close to the op- Based on the analysis, we develop an efficient tool to auto- timum, warm start is very useful.
    [Show full text]
  • Neural Networks and Backpropagation
    CS 179: LECTURE 14 NEURAL NETWORKS AND BACKPROPAGATION LAST TIME Intro to machine learning Linear regression https://en.wikipedia.org/wiki/Linear_regression Gradient descent https://en.wikipedia.org/wiki/Gradient_descent (Linear classification = minimize cross-entropy) https://en.wikipedia.org/wiki/Cross_entropy TODAY Derivation of gradient descent for linear classifier https://en.wikipedia.org/wiki/Linear_classifier Using linear classifiers to build up neural networks Gradient descent for neural networks (Back Propagation) https://en.wikipedia.org/wiki/Backpropagation REFRESHER ON THE TASK Note “Grandmother Cell” representation for {x,y} pairs. See https://en.wikipedia.org/wiki/Grandmother_cell REFRESHER ON THE TASK Find i for zi: “Best-index” -- estimated “Grandmother Cell” Neuron Can use parallel GPU reduction to find “i” for largest value. LINEAR CLASSIFIER GRADIENT We will be going through some extra steps to derive the gradient of the linear classifier -- We’ll be using the “Softmax function” https://en.wikipedia.org/wiki/Softmax_function Similarities will be seen when we start talking about neural networks LINEAR CLASSIFIER J & GRADIENT LINEAR CLASSIFIER GRADIENT LINEAR CLASSIFIER GRADIENT LINEAR CLASSIFIER GRADIENT GRADIENT DESCENT GRADIENT DESCENT, REVIEW GRADIENT DESCENT IN ND GRADIENT DESCENT STOCHASTIC GRADIENT DESCENT STOCHASTIC GRADIENT DESCENT STOCHASTIC GRADIENT DESCENT, FOR W LIMITATIONS OF LINEAR MODELS Most real-world data is not separable by a linear decision boundary Simplest example: XOR gate What if we could combine the results of multiple linear classifiers? Combine two OR gates with an AND gate to get a XOR gate ANOTHER VIEW OF LINEAR MODELS NEURAL NETWORKS NEURAL NETWORKS EXAMPLES OF ACTIVATION FNS Note that most derivatives of tanh function will be zero! Makes for much needless computation in gradient descent! MORE ACTIVATION FUNCTIONS https://medium.com/@shrutijadon10104776/survey-on- activation-functions-for-deep-learning-9689331ba092 Tanh and sigmoid used historically.
    [Show full text]
  • Lecture 2: Linear Classifiers
    Lecture 2: Linear Classifiers Andr´eMartins Deep Structured Learning Course, Fall 2018 Andr´eMartins (IST) Lecture 2 IST, Fall 2018 1 / 117 Course Information • Instructor: Andr´eMartins ([email protected]) • TAs/Guest Lecturers: Erick Fonseca & Vlad Niculae • Location: LT2 (North Tower, 4th floor) • Schedule: Wednesdays 14:30{18:00 • Communication: piazza.com/tecnico.ulisboa.pt/fall2018/pdeecdsl Andr´eMartins (IST) Lecture 2 IST, Fall 2018 2 / 117 Announcements Homework 1 is out! • Deadline: October 10 (two weeks from now) • Start early!!! List of potential projects will be sent out soon! • Deadline for project proposal: October 17 (three weeks from now) • Teams of 3 people Andr´eMartins (IST) Lecture 2 IST, Fall 2018 3 / 117 Today's Roadmap Before talking about deep learning, let us talk about shallow learning: • Supervised learning: binary and multi-class classification • Feature-based linear classifiers • Rosenblatt's perceptron algorithm • Linear separability and separation margin: perceptron's mistake bound • Other linear classifiers: naive Bayes, logistic regression, SVMs • Regularization and optimization • Limitations of linear classifiers: the XOR problem • Kernel trick. Gaussian and polynomial kernels. Andr´eMartins (IST) Lecture 2 IST, Fall 2018 4 / 117 Fake News Detection Task: tell if a news article / quote is fake or real. This is a binary classification problem. Andr´eMartins (IST) Lecture 2 IST, Fall 2018 5 / 117 Fake Or Real? Andr´eMartins (IST) Lecture 2 IST, Fall 2018 6 / 117 Fake Or Real? Andr´eMartins (IST) Lecture 2 IST, Fall 2018 7 / 117 Fake Or Real? Andr´eMartins (IST) Lecture 2 IST, Fall 2018 8 / 117 Fake Or Real? Andr´eMartins (IST) Lecture 2 IST, Fall 2018 9 / 117 Fake Or Real? Can a machine determine this automatically? Can be a very hard problem, since fact-checking is hard and requires combining several knowledge sources ..
    [Show full text]
  • Hyperplane Based Classification: Perceptron and (Intro
    Hyperplane based Classification: Perceptron and (Intro to) Support Vector Machines Piyush Rai CS5350/6350: Machine Learning September 8, 2011 (CS5350/6350) Hyperplane based Classification September8,2011 1/20 Hyperplane Separates a D-dimensional space into two half-spaces Defined by an outward pointing normal vector w RD ∈ (CS5350/6350) Hyperplane based Classification September8,2011 2/20 Hyperplane Separates a D-dimensional space into two half-spaces Defined by an outward pointing normal vector w RD ∈ w is orthogonal to any vector lying on the hyperplane (CS5350/6350) Hyperplane based Classification September8,2011 2/20 Hyperplane Separates a D-dimensional space into two half-spaces Defined by an outward pointing normal vector w RD ∈ w is orthogonal to any vector lying on the hyperplane Assumption: The hyperplane passes through origin. (CS5350/6350) Hyperplane based Classification September8,2011 2/20 Hyperplane Separates a D-dimensional space into two half-spaces Defined by an outward pointing normal vector w RD ∈ w is orthogonal to any vector lying on the hyperplane Assumption: The hyperplane passes through origin. If not, have a bias term b; we will then need both w and b to define it b > 0 means moving it parallely along w (b < 0 means in opposite direction) (CS5350/6350) Hyperplane based Classification September8,2011 2/20 Linear Classification via Hyperplanes Linear Classifiers: Represent the decision boundary by a hyperplane w For binary classification, w is assumed to point towards the positive class (CS5350/6350) Hyperplane based Classification
    [Show full text]
  • 2 Linear Classifiers and Perceptrons
    Linear Classifiers and Perceptrons 7 2 Linear Classifiers and Perceptrons CLASSIFIERS You are given sample of n observations, each with d features [aka predictors]. Some observations belong to class C; some do not. Example: Observations are bank loans Features are income & age (d = 2) Some are in class “defaulted,” some are not Goal: Predict whether future borrowers will default, based on their income & age. Represent each observation as a point in d-dimensional space, called a sample point / a feature vector / independent variables. overfitting X X X X X X X C X X X X X C X C C X C C X C X C C C income income income X C C X C X C X X X C C X C C C X C C C C C age age age [Draw this by hand; decision boundaries last. classify3.pdf ] [We draw these lines/curves separating C’s from X’s. Then we use these curves to predict which future borrowers will default. In the last example, though, we’re probably overfitting, which could hurt our predic- tions.] decision boundary: the boundary chosen by our classifier to separate items in the class from those not. overfitting: When sinuous decision boundary fits sample points so well that it doesn’t classify future points well. [A reminder that underlined phrases are definitions, worth memorizing.] Some (not all) classifiers work by computing a decision function: A function f (x) that maps a point x to a scalar such that f (x) > 0 if x class C; 2 f (x) 0 if x < class C.
    [Show full text]
  • Machine Learning Empirical Risk Minimization
    Machine Learning Empirical Risk Minimization Dmitrij Schlesinger WS2014/2015, 08.12.2014 Recap – tasks considered before n For a training dataset L = (xl, yl) ... with xl ∈ R (data) and yl ∈ {+1, −1} (classes) fing a separating hyperplane yl · [hw, xli + b] ≥ 0 ∀l → Perceptron algorithm The goal is to fing a "corridor" (stripe) of the maximal width that separates the data → Large Margin learning, linear SVM 1 kwk2 → min 2 w s.t. yl[hw, xli + b] ≥ 1 In both cases the data is assumed to be separable. What if not? ML: Empirical Risk Minimization 08.12.2014 2 Empirical Risk Minimization Let a loss function C(y, y0) be given that penalizes deviations between the true class and the estimated one (like the loss in the Bayesian Decision theory). The Empirical Risk of a decision strategy is the total loss over the training set: X R(e) = C yl, e(xl) → min e l It should bi minimized with respect to the decision strategy e. Special case (today): – the set of decisions is {+1, −1} – the loss is the delta-function C(y, y0) = δ(y 6= y0) – the decision strategy can be expressed in the form e(x) = sign f(x) with an evaluation function f : X → R Example: f(x) = hw, xi − b is a linear classifier. ML: Empirical Risk Minimization 08.12.2014 3 Hinge Loss The problem: the subject is not convex The way out: replace the real loss by its convex upper bound ← example for y = 1 (for y = −1 it should be flipped) δ y 6= sign f(x) ≤ max 0, 1 − y · f(x) It is called Hinge Loss ML: Empirical Risk Minimization 08.12.2014 4 Sub-gradient algorithm Let the evaluation function be parameterized, i.e.
    [Show full text]
  • Using Linear Classifier Probes
    Under review as a conference paper at ICLR 2017 UNDERSTANDING INTERMEDIATE LAYERS USING LINEAR CLASSIFIER PROBES Guillaume Alain & Yoshua Bengio Department of Computer Science and Operations Research Universite´ de Montreal´ Montreal, QC. H3C 3J7 [email protected] ABSTRACT Neural network models have a reputation for being black boxes. We propose a new method to better understand the roles and dynamics of the intermediate layers. This has direct consequences on the design of such models and it enables the expert to be able to justify certain heuristics (such as adding auxiliary losses in middle layers). Our method uses linear classifiers, referred to as “probes”, where a probe can only use the hidden units of a given intermediate layer as discriminating features. Moreover, these probes cannot affect the training phase of a model, and they are generally added after training. They allow the user to visualize the state of the model at multiple steps of training. We demonstrate how this can be used to develop a better intuition about models and to diagnose potential problems. 1 INTRODUCTION The recent history of deep neural networks features an impressive number of new methods and technological improvements to allow the training of deeper and more powerful networks. Despite this, models still have a reputation for being black boxes. Neural networks are criticized for their lack of interpretability, which is a tradeoff that we accept because of their amazing performance on many tasks. Efforts have been made to identify the role played by each layer, but it can be hard to find a meaning to individual layers.
    [Show full text]
  • Chapter 2 Classification
    Chapter 2 Classification In this chapter we study one of the most basic topics in machine learning called classi- fication. By classification we meant a supervised learning procedure where we try to classify a sample based on a model learned from the data in a training dataset. The goal of this chapter is to understand the general principle of classification, to explore various methodologies so that we can do classification, and to make connections between different methods. 2.1 Discriminant Function Basic Terminologies A classification method always starts with a pair of variables (x; y). The first variable x 2 X ⊂ Rd is an input vector. The set X is called the input space. Depending on the application, an input could the intensity of a pixel, a wavelet coefficient, the phase angle of wave, or anything we want to model. We assume that each input vector has a dimensionality d and d is finite. The second variable y 2 f1; : : : ; kg = Y denotes the class label of the classes fC1;:::; Ckg. The choice of the class labels is arbitrary. They do not need to be positive integers. For example, in binary classification, one can choose y 2 f0; 1g or y 2 {−1; +1g, depending which one is more convenient mathematically. In supervised learning, we assume that we have a training set D. A training set is a collection of paired variables (xj; yj), for j = 1; : : : ; n. The vector xj denotes the j-th sample input in D, and yj denotes the corresponding class label. The relationship between xj and yj is specified by the target function f : X!Y such that yj = f(xj).
    [Show full text]
  • Linear Classification Overview Logistic Regression Model I Logistic
    Overview I Logistic regression model I Linear classifiers I Gradient descent and SGD Linear classification I Multinomial logistic regression model COMS 4771 Fall 2019 0 / 27 1 / 27 Logistic regression model I Logistic regression model II d t I Suppose x is given by d real-valued features, so x R , while I Sigmoid function σ(t) := 1/(1 + e− ) ∈ y 1, +1 . I Useful property: 1 σ(t) = σ( t) ∈ {− } − − I Logistic regression model for (X,Y ): I Pr(Y = +1 X = x) = σ(xTw) | T T I Y X = x is Bernoulli (but taking values in 1, +1 rather than I Pr(Y = 1 X = x) = 1 σ(x w) = σ( x w) | {− } − | − − 0, 1 ), with “heads probability” parameter I Convenient formula: for each y 1, +1 , { } ∈ {− } 1 . Pr(Y = y X = x) = σ(yxTw). 1 + exp( xTw) | − d I w R is parameter vector of interest 1 ∈ I w not involved in marginal distribution of X (which we don’t care 0.8 much about) 0.6 0.4 0.2 0 -6 -4 -2 0 2 4 6 Figure 1: Sigmoid function 2 / 27 3 / 27 Log-odds in logistic regression model Optimal classifier in logistic regression model I Log-odds in the model is given by a linear function: I Recall that Bayes classifier is Pr(Y = +1 X = x) T ln | = x w. ? +1 if Pr(Y = +1 X = x) > 1/2 Pr(Y = 1 X = x) f (x) = | − | 1 otherwise. (− I Just like in linear regression, common to use feature expansion! d+1 If distribution of (X,Y ) comes from logistic regression model with I E.g., affine feature expansion ϕ(x) = (1, x) R I ∈ parameter w, then Bayes classifier is +1 if xTw > 0 f ?(x) = 1 otherwise.
    [Show full text]
  • Differentially Private Empirical Risk Minimization
    JournalofMachineLearningResearch12(2011)1069-1109 Submitted 6/10; Revised 2/11; Published 3/11 Differentially Private Empirical Risk Minimization Kamalika Chaudhuri [email protected] Department of Computer Science and Engineering University of California, San Diego La Jolla, CA 92093, USA Claire Monteleoni [email protected] Center for Computational Learning Systems Columbia University New York, NY 10115, USA Anand D. Sarwate [email protected] Information Theory and Applications Center University of California, San Diego La Jolla, CA 92093-0447, USA Editor: Nicolas Vayatis Abstract Privacy-preserving machine learning algorithms are crucial for the increasingly common setting in which personal data, such as medical or financial records, are analyzed. We provide general techniques to produce privacy-preserving approximations of classifiers learned via (regularized) empirical risk minimization (ERM). These algorithms are private under the ε-differential privacy definition due to Dwork et al. (2006). First we apply the output perturbation ideas of Dwork et al. (2006), to ERM classification. Then we propose a new method, objective perturbation, for privacy-preserving machine learning algorithm design. This method entails perturbing the objective function before optimizing over classifiers. If the loss and regularizer satisfy certain convexity and differentiability criteria, we prove theoretical results showing that our algorithms preserve privacy, and provide generalization bounds for linear and nonlinear kernels. We further present a privacy- preserving technique for tuning the parameters in general machine learning algorithms, thereby providing end-to-end privacy guarantees for the training process. We apply these results to produce privacy-preserving analogues of regularized logistic regression and support vector machines. We obtain encouraging results from evaluating their performance on real demographic and benchmark data sets.
    [Show full text]
  • Tutorial on Support Vector Machine (SVM) Vikramaditya Jakkula, School of EECS, Washington State University, Pullman 99164
    Tutorial on Support Vector Machine (SVM) Vikramaditya Jakkula, School of EECS, Washington State University, Pullman 99164. Abstract: In this tutorial we present a brief introduction to SVM, and we discuss about SVM from published papers, workshop materials & material collected from books and material available online on the World Wide Web. In the beginning we try to define SVM and try to talk as why SVM, with a brief overview of statistical learning theory. The mathematical formulation of SVM is presented, and theory for the implementation of SVM is briefly discussed. Finally some conclusions on SVM and application areas are included. Support Vector Machines (SVMs) are competing with Neural Networks as tools for solving pattern recognition problems. This tutorial assumes you are familiar with concepts of Linear Algebra, real analysis and also understand the working of neural networks and have some background in AI. Introduction Machine Learning is considered as a subfield of Artificial Intelligence and it is concerned with the development of techniques and methods which enable the computer to learn. In simple terms development of algorithms which enable the machine to learn and perform tasks and activities. Machine learning overlaps with statistics in many ways. Over the period of time many techniques and methodologies were developed for machine learning tasks [1]. Support Vector Machine (SVM) was first heard in 1992, introduced by Boser, Guyon, and Vapnik in COLT-92. Support vector machines (SVMs) are a set of related supervised learning methods used for classification and regression [1]. They belong to a family of generalized linear classifiers. In another terms, Support Vector Machine (SVM) is a classification and regression prediction tool that uses machine learning theory to maximize predictive accuracy while automatically avoiding over-fit to the data.
    [Show full text]
  • 2DI70 - Statistical Learning Theory Lecture Notes
    2DI70 - Statistical Learning Theory Lecture Notes Rui Castro April 3, 2018 Some of the material in these notes will be published by Cambridge University Press as Statistical Machine Learning: A Gentle Primer by Rui M. Castro and Robert D. Nowak. This early draft is free to view and download for personal use only. Not for re-distribution, re-sale or use in derivative works. c Rui M. Castro and Robert D. Nowak, 2017. 2 Contents Contents 2 1 Introduction 9 1.1 Learning from Data..................................9 1.1.1 Data Spaces...................................9 1.1.2 Loss Functions................................. 10 1.1.3 Probability Measure and Expectation.................... 11 1.1.4 Statistical Risk................................. 11 1.1.5 The Learning Problem............................. 12 2 Binary Classification and Regression 15 2.1 Binary Classification.................................. 15 2.2 Regression........................................ 18 2.3 Empirical Risk Minimization............................. 19 2.4 Model Complexity and Overfitting.......................... 21 2.5 Exercises........................................ 25 3 Competing Goals: approximation vs. estimation 29 3.1 Strategies To Avoid Overfitting............................ 30 3.1.1 Method of Sieves................................ 31 3.1.2 Complexity Penalization Methods...................... 32 3.1.3 Hold-out Methods............................... 33 4 Estimation of Lipschitz smooth functions 37 4.1 Setting.......................................... 37 4.2 Analysis........................................
    [Show full text]