DOCSLIB.ORG

Linear Models for Classification

Total Page：16

File Type：pdf, Size：1020Kb

Download full-text PDF Read full-text
Linear Models for Classification CS480 Introduction to Machine Learning Linear Models for Classification Edith Law Based on slides from Joelle Pineau Probabilistic Framework for Classification As we saw in the last lecture, learning can be posed a problem of statistical inference. The premise is that for any learning problem, if we have access to the underlying probability distribution D, then we could form a Bayes optimal classifier as: f (BO)(x)̂ = arg max D(x,̂ y)̂ y∈̂ Y But since we don't have access to D, we can instead estimate this distribution from data. The probabilistic framework allows you to choose a model to represent your data, and develop algorithms that have inductive biases that are closer to what you, as a designer, believe. 2 Classification by Density Estimation The most direct way to construct such a probability distribution is to select a family of parametric distributions. For example, we can select a Gaussian (or Normal) distribution. Gaussian is parametric: its parameters are its mean and variance. The job of learning is to infer which parameters are “best” in terms of describing the observed training data. In density estimation, we assume that the sample data is drawn independently from the same distribution D. That is, your nth sample (xn, yn) from D does not depend on the previous n-1 samples. This is called the i.i.d. (independently and identically distributed) assumption. 3 Consider the Problem of Flipping a Biased Coin You are flipping a coin, and you want to find out if it is biased. This coin has some unknown probability of β of coming up Head (H), and some probability ( 1 − β ) of coming up Tail (T). Suppose you flip the coin and observed HHTH. Assuming that each flip is independent, the probability of the data (i.e., the coin flip sequence) is p(D|β) = p(HHTH|β) = β ⋅ β ⋅ (1 − β) ⋅ β = β3(1 − β) = β3 − β4 4 Statistical Estimation To find the most likely value of β , you want to select parameters to maximize the the probability of the data. This means taking the derivative of p ( D | β ) with respect to β , set it to zero and solve for δ [β3 − β4] = 3β2 − 4β2 δβ 4β3 = 3β2 ⟺ 4β = 3 3 ⟺ β = 4 This gives you the maximum likelihood estimate of the parameters of your model. 5 Statistical Estimation But notice that the likelihood of the data p ( D | β ) actually has the form of the binomial distribution: βH(1 − β)T The log likelihood of the binomial distribution can be rewritten log p(D|β) = log βH(1 − β)T = H log β + T log(1 − β) When we differentiate with respect to beta, and set to zero. δ H T p(D|β) = − δβ β 1 − β H ⇒ β = The estimate is equivalent to the fraction of H + T the observed data that comes up heads! Maximum Likelihood Estimate is nothing more than the relative frequency of observing a certain event (e.g., observing heads). 6 From Estimation to Inference likelihood prior probability P(e|H)P(H) P(H|e) = P(e) posterior probability normalizing constant Maximum likelihood Maximum a Posteriori Bayesian Estimate P(e|hi)P(hi) hML = arg max P(e|hi) hMAP = arg max P(hi |e) P(hi |e) = hi hi P(e) P(X|e) ≈ P(X|hML) P(X|e) ≈ P(X|hMAP) P(X|e) = ∑ P(X|hi) P(hi |e) i 7 Classification Given data set D=<xi,yi>, i=1:n, with discrete yi, find a hypothesis which “best fits” the data. If yi ∈ {0, 1} this is binary classification. If yi can take more than two values, the problem is called multi-class classification. 8 Applications for Classification • Text classification (spam filtering, news filtering, building web dir, etc.) • Image classification (face detection, object recognition, etc.) • Prediction of cancer recurrence. • Financial forecasting. • Many, many more! 9 Simple Example • Given “nucleus size”, predict cancer recurrence. • Univariate input: X = nucleus size. • Binary output: Y = {N = 0; R = 1} • Try: Minimize the least-square error. N=NoRecurrence R=Recurrence 10 Predicting a Class from Linear Regression Here red line is: Y’ = X (XTX)-1 XTY How to get a binary output? •Threshold the output: { y <= t for NoRecurrence, y > t for Recurrence} •Interpret output as probability: y = P(Recurrence) In the probabilistic framework, the goal is to learn/estimate the posterior distribution P(y|x) and using it for classification. 11 Models for Binary Classification Two probabilistic approaches: 1. Generative learning 2. Discriminative learning 12 Models for Binary Classification Two probabilistic approaches: 1. Generative learning 2. Discriminative learning 13 Generative Learning Separately models P(x|y) and P(y), use these, through Bayes rule to estimate P(y|x). 14 Considering Binary Classification Problems Suppose you are trying to predict whether a movie review is positive based on the words that appear in the review. p(y = + |x) Generative learning says that to calculate P(y|x), we estimate the likelihood P(x|y) and the prior P(y) and multiply them together. P(x|y) = P(x1, x2, …, xm |y) = P(x1 |y) P(x2 |y, x1) P(x3 |y, x1, x2), …, P(xm |y, x1, x2, …, xm−1) (from general rules of probabilities) The challenge is that this likelihood distribution is over a lot of parameters. 15 Naive Bayes Assumption Assume the xj are conditionally independent given y, i.e., p(xj |y, xk) = p(xj |y), ∀j, k Example: Thunder is conditionally independent of Rain given Lightning. Lightning Thunder Rain P(Thunder |Rain, Lightning) = P(Thunder |Lightning) 16 Naive Bayes Assumption Conditional Independence Assumption: the probability distribution governing a particular feature is conditionally independent to the probability distribution governing another feature, as well as any combinations/subsets of other features, given that we know the label. y x1 x2 x3 … xm 17 Naive Bayes Assumption p(xj |y, xk) = p(xj |y), ∀j, k This greatly simplifies the computation of the likelihood: P(x|y) = P(x1, x2, …, xm |y) = P(x1 |y) P(x2 |y, x1) P(x3 |y, x1, x2), …, P(xm |y, x1, x2, …, xm−1) (from general rules of probabilities) M = P(x1 |y) P(x2 |y) P(x3 |y)…P(xm |y) = ∏P(xi |y) (from the Naïve Bayes assumption) i=1 18 Naive Bayes Assumption y x1 x2 x3 … xm How many parameters to estimate? Assume m binary features. •without NB assumption: 2(2m-1)=O(2m) numbers to describe model. •with NB assumption: 2m=O(m) numbers to describe model. Useful when the number of features is high. 19 Training a Naive Bayes Classifier Learning = Find parameters that maximize the log-likelihood: n m P(y|x) ∝ P(y)P(x|y) = ∏(P(yi)∏P(xi,j |yi)) i=1 j=1 •Samples i are independent, so we take product over n. •Input features are independent (cond. on y) so we take product over m. Suppose that x, y are binary variables, m=1. y Define: θ1 = P(y = 1) θj,1 = P(xj = 1|y = 1) θj,0 = P(xj = 1|y = 0) x 20 Training a Naive Bayes Classifier Likelihood of the output variable: y 1−y L(θ1) = θ1(1 − θ1) Likelihood for all parameters: n m log L(θ1, θi,1, θi,0 |D) = ∑ [ log P(yi) + ∑ log P(xi,j |yi) ] i j=1 n = ∑ [ yi log θ1 + (1 − yi) log(1 − θ1) ] i m +∑ yi(xi,j log θi,1 + (1 − xi,j)log(1 − θi,1)) j=1 m +∑ (1 − yi)(xi,j log θi,0 + (1 − xi,j)log(1 − θi,0) j=1 The log likelihood will have other form if params P(x|y) have other form. In this case, we chose Bernouilli. If the data is continuous, we can choose Gaussian. You can choose the distribution based on your knowledge of the problem. The choice of distribution is a form of inductive bias. 21 Training a Naive Bayes Classifier To estimate θ 1 , take derivative of the log likelihood θ 1 , set to 0 and solve: n δL yi 1 − yi = ∑ ( − ) = 0 δθ1 i=1 θ1 1 − θ1 1 n We get: number of examples where y=1 θ1 = ∑ yi = n i number of examples Similarly, we get: number of examples where xj=1 and y=1 θj,1 = frequencies of number of examples where y=1 events! number of examples where x =1 and y=0 θ = j j,0 number of examples where y=0 22 Log Likelihood Ratio Suppose we have 2 classes: y ∊ {0, 1} What is the probability of a given input x having class y = 1? Consider Bayes rule: P(x, y = 1) P(x|y = 1)P(y = 1) P(y = 1|x) = = P(x) P(x|y = 1)P(y = 1) + P(x|y = 0)p(y = 0) 1 1 1 = = P(x|y = 0)P(y = 0) P(x|y = 0)P(y = 0) = 1 + exp ln 1 + exp(−a) 1 + P(x|y = 1)P(y = 1) ( P(x|y = 1)P(y = 1) ) where P(x|y = 1)P(y = 1) P(y = 1|x) (By Bayes rule; P(x) on top and a = ln = ln P(x|y = 0)P(y = 0) P(y = 0|x) bottom cancels out.) � is the log-odds ratio of data being class 1 vs. class 0. It tells us which class (y=1 vs y=0) has higher probability and we can classify x accordingly. 23 Naive Bayes Decision Boundary What does the decision boundary look like for Naive Bayes? Let’s start from the log-odds ratio: P(y = 1|x) P(x|y = 1)P(y = 1) log = log P(y = 0|x) P(x|y = 0)P(y = 0) m P(y = 1) ∏j=1 P(xj |y = 1) = log + log m P(y = 0) ∏j=1 P(xj |y = 0) m P(y = 1) P(xj |y = 1) = log + ∑ log P(y = 0) j=1 P(xj |y = 0) 24 Naive Bayes Decision Boundary Consider the case where features are binary: xj = {0, 1} Define: P(xj = 0|y = 1) P(xj = 1|y = 1) wj,0 = log wj,1 = log P(xj = 0|y = 0) P(xj = 1|y = 0) Now we have: m P(y = 1|x) P(y = 1) P(xj |y = 1) log = log + ∑ log P(y = 0|x) P(y = 0) j=1 P(xj |y = 0) P(y = 1) m = log + ∑ (wj,0(1 − xj) + wj,1xj) P(y = 0) j=1 P(y = 1) m m = log + ∑ wj,0 + ∑ (wj,1 − wj,0)xj P(y = 0) j=1 j=1 This is a linear decision boundary! w0 wTx 25 Naive Bayes Decision Boundary Results show that Naive Bayes with binary features has precisely the form of a linear model.
Load more

Recommended publications
  • Confidence Intervals for Probabilistic Network Classifiers
    Computational Statistics & Data Analysis 49 (2005) 998–1019 www.elsevier.com/locate/csda Confidence intervals for probabilistic network classifiersଁ M. Egmont-Petersena,∗, A. Feeldersa, B. Baesensb aUtrecht University, Institute of Information and Computing Sciences, P. O. Box 80.089, 3508, TB Utrecht, The Netherlands bUniversity of Southampton, School of Management, UK Received 12 July 2003; received in revised form 28 June 2004 Available online 25 July 2004 Abstract Probabilistic networks (Bayesian networks) are suited as statistical pattern classifiers when the feature variables are discrete. It is argued that their white-box character makes them transparent, a requirement in various applications such as, e.g., credit scoring. In addition, the exact error rate of a probabilistic network classifier can be computed without a dataset. First, the exact error rate for probabilistic network classifiers is specified. Secondly, the exact sampling distribution for the conditional probability estimates in a probabilistic network classifier is derived. Each conditional probability is distributed according to the bivariate binomial distribution. Subsequently, an approach for computing the sampling distribution and hence confidence intervals for the posterior probability in a probabilistic network classifier is derived. Our approach results in parametric bootstrap confidence intervals. Experiments with general probabilistic network classifiers, the Naive Bayes classifier and tree augmented Naive Bayes classifiers (TANs) show that our approximation performs well. Also simulations performed with the Alarm network show good results for large training sets. The amount of computation required is exponential in the number of feature variables. For medium and large-scale classification problems, our approach is well suited for quick simulations.A running example from the domain of credit scoring illustrates how to actually compute the sampling distribution of the posterior probability.
    [Show full text]
  • Bayesian Learning
    SC4/SM8 Advanced Topics in Statistical Machine Learning Bayesian Learning Dino Sejdinovic Department of Statistics Oxford Slides and other materials available at: http://www.stats.ox.ac.uk/~sejdinov/atsml/ Department of Statistics, Oxford SC4/SM8 ATSML, HT2018 1 / 14 Bayesian Learning Review of Bayesian Inference The Bayesian Learning Framework Bayesian learning: treat parameter vector θ as a random variable: process of learning is then computation of the posterior distribution p(θjD). In addition to the likelihood p(Djθ) need to specify a prior distribution p(θ). Posterior distribution is then given by the Bayes Theorem: p(Djθ)p(θ) p(θjD) = p(D) Likelihood: p(Djθ) Posterior: p(θjD) Prior: p(θ) Marginal likelihood: p(D) = Θ p(Djθ)p(θ)dθ ´ Summarizing the posterior: MAP Posterior mode: θb = argmaxθ2Θ p(θjD) (maximum a posteriori). mean Posterior mean: θb = E [θjD]. Posterior variance: Var[θjD]. Department of Statistics, Oxford SC4/SM8 ATSML, HT2018 2 / 14 Bayesian Learning Review of Bayesian Inference Bayesian Inference on the Categorical Distribution Suppose we observe the with yi 2 f1;:::; Kg, and model them as i.i.d. with pmf π = (π1; : : : ; πK): n K Y Y nk p(Djπ) = πyi = πk i=1 k=1 Pn PK with nk = i=1 1(yi = k) and πk > 0, k=1 πk = 1. The conjugate prior on π is the Dirichlet distribution Dir(α1; : : : ; αK) with parameters αk > 0, and density PK K Γ( αk) Y p(π) = k=1 παk−1 QK k k=1 Γ(αk) k=1 PK on the probability simplex fπ : πk > 0; k=1 πk = 1g.
    [Show full text]
  • Bayesian Machine Learning
    Advanced Topics in Machine Learning: Bayesian Machine Learning Tom Rainforth Department of Computer Science Hilary 2020 Contents 1 Introduction 1 1.1 A Note on Advanced Sections . .3 2 A Brief Introduction to Probability 4 2.1 Random Variables, Outcomes, and Events . .4 2.2 Probabilities . .4 2.3 Conditioning and Independence . .5 2.4 The Laws of Probability . .6 2.5 Probability Densities . .7 2.6 Expectations and Variances . .8 2.7 Measures [Advanced Topic] . 10 2.8 Change of Variables . 12 3 Machine Learning Paradigms 13 3.1 Learning From Data . 13 3.2 Discriminative vs Generative Machine Learning . 16 3.3 The Bayesian Paradigm . 20 3.4 Bayesianism vs Frequentism [Advanced Topic] . 23 3.5 Further Reading . 31 4 Bayesian Modeling 32 4.1 A Fundamental Assumption . 32 4.2 The Bernstein-Von Mises Theorem . 34 4.3 Graphical Models . 35 4.4 Example Bayesian Models . 38 4.5 Nonparametric Bayesian Models . 42 4.6 Gaussian Processes . 43 4.7 Further Reading . 52 5 Probabilistic Programming 53 5.1 Inverting Simulators . 54 5.2 Differing Approaches . 58 5.3 Bayesian Models as Program Code [Advanced Topic] . 62 5.4 Further Reading . 73 Contents iii 6 Foundations of Bayesian Inference and Monte Carlo Methods 74 6.1 The Challenge of Bayesian Inference . 74 6.2 Deterministic Approximations . 77 6.3 Monte Carlo . 79 6.4 Foundational Monte Carlo Inference Methods . 83 6.5 Further Reading . 93 7 Advanced Inference Methods 94 7.1 The Curse of Dimensionality . 94 7.2 Markov Chain Monte Carlo . 97 7.3 Variational Inference .
    [Show full text]
  • Agnostic Bayes
    Agnostic Bayes Thèse Alexandre Lacoste Doctorat en informatique Philosophiæ doctor (Ph.D.) Québec, Canada © Alexandre Lacoste, 2015 Résumé L’apprentissage automatique correspond à la science de l’apprentissage à partir d’exemples. Des algorithmes basés sur cette approche sont aujourd’hui omniprésents. Bien qu’il y ait eu un progrès significatif, ce domaine présente des défis importants. Par exemple, simplement sélectionner la fonction qui correspond le mieux aux données observées n’offre aucune ga- rantie statistiques sur les exemples qui n’ont pas encore été observées. Quelques théories sur l’apprentissage automatique offrent des façons d’aborder ce problème. Parmi ceux-ci, nous présentons la modélisation bayésienne de l’apprentissage automatique et l’approche PAC- bayésienne pour l’apprentissage automatique dans une vue unifiée pour mettre en évidence d’importantes similarités. Le résultat de cette analyse suggère que de considérer les réponses de l’ensemble des modèles plutôt qu’un seul correspond à un des éléments-clés pour obtenir une bonne performance de généralisation. Malheureusement, cette approche vient avec un coût de calcul élevé, et trouver de bonnes approximations est un sujet de recherche actif. Dans cette thèse, nous présentons une approche novatrice qui peut être appliquée avec un faible coût de calcul sur un large éventail de configurations d’apprentissage automatique. Pour atteindre cet objectif, nous appliquons la théorie de Bayes d’une manière différente de ce qui est conventionnellement fait pour l’apprentissage automatique. Spécifiquement, au lieu de chercher le vrai modèle à l’origine des données observées, nous cherchons le meilleur modèle selon une métrique donnée. Même si cette différence semble subtile, dans cette approche, nous ne faisons pas la supposition que le vrai modèle appartient à l’ensemble de modèles explorés.
    [Show full text]
  • On Discriminative Bayesian Network Classifiers and Logistic Regression
    Machine Learning, 59, 267–296, 2005 2005 Springer Science + Business Media, Inc. Manufactured in The Netherlands. On Discriminative Bayesian Network Classifiers and Logistic Regression TEEMU ROOS teemu.roos@hiit.fi HANNES WETTIG Complex Systems Computation Group, Helsinki Institute for Information Technology, P.O. Box 9800, FI-02015 TKK, Finland PETER GRUNWALD¨ Centrum voor Wiskunde en Informatica, P.O. Box 94079, NL-1090 GB Amsterdam, The Netherlands PETRI MYLLYMAKI¨ HENRY TIRRI Complex Systems Computation Group, Helsinki Institute for Information Technology, P.O. Box 9800, FI-02015 TKK, Finland Editors: Pedro Larranaga,˜ Jose A. Lozano, Jose M. Pena˜ and Inaki˜ Inza Abstract. Discriminative learning of the parameters in the naive Bayes model is known to be equivalent to a logistic regression problem. Here we show that the same fact holds for much more general Bayesian network models, as long as the corresponding network structure satisfies a certain graph-theoretic property. The property holds for naive Bayes but also for more complex structures such as tree-augmented naive Bayes (TAN) as well as for mixed diagnostic-discriminative structures. Our results imply that for networks satisfying our property, the conditional likelihood cannot have local maxima so that the global maximum can be found by simple local optimization methods. We also show that if this property does not hold, then in general the conditional likelihood can have local, non-global maxima. We illustrate our theoretical results by empirical experiments with local optimization in a conditional naive Bayes model. Furthermore, we provide a heuristic strategy for pruning the number of parameters and relevant features in such models.
    [Show full text]
  • Areview of Bayesian Statistical Approaches for Big Data
    AREVIEW OF BAYESIAN STATISTICAL APPROACHES FOR BIG DATA APREPRINT Farzana Jahan∗ Insha Ullah School of Mathematical Sciences School of Mathematical Sciences ARC Centre of Mathematical and Statistical Frontiers ARC Centre of Mathematical and Statistical Frontiers Science and Engineering Faculty Science and Engineering Faculty Queensland University of Technology Queensland University of Technology Brisbane, Queensland, Australia Brisbane, Queensland, Australia [email protected] [email protected] Kerrie L. Mengersen School of Mathematical Sciences ARC Centre of Mathematical and Statistical Frontiers Science and Engineering Faculty Queensland University of Technology Brisbane, Queensland, Australia k.mengersen @qut.edu.au June 9, 2020 ABSTRACT The modern era is characterised as an era of information or Big Data. This has motivated a huge literature on new methods for extracting information and insights from these data. A natural question is how these approaches differ from those that were available prior to the advent of Big Data. We present a review of published studies that present Bayesian statistical approaches specifically for Big Data and discuss the reported and perceived benefits of these approaches. We conclude by addressing the question of whether focusing only on improving computational algorithms and infrastructure will be enough to face the challenges of Big Data. Keywords Bayesian Statistics · Bayesian modelling · Bayesian computation · Scalable algorithms 1 Introduction arXiv:2006.04565v1 [stat.CO] 8 Jun 2020 Although there are many variations on the definition of Big Data [1, 2, 3, 4], it is clear that it encompasses large and often diverse quantitative data obtained from increasing numerous sources at different individual, spatial and temporal scales, and with different levels of quality.
    [Show full text]
  • Bayesian Statistical Inference for Intractable Likelihood Models Louis Raynal
    Bayesian statistical inference for intractable likelihood models Louis Raynal To cite this version: Louis Raynal. Bayesian statistical inference for intractable likelihood models. Statistics [math.ST]. Université Montpellier, 2019. English. NNT : 2019MONTS035. tel-02445937 HAL Id: tel-02445937 https://tel.archives-ouvertes.fr/tel-02445937 Submitted on 20 Jan 2020 HAL is a multi-disciplinary open access L’archive ouverte pluridisciplinaire HAL, est archive for the deposit and dissemination of sci- destinée au dépôt et à la diffusion de documents entific research documents, whether they are pub- scientifiques de niveau recherche, publiés ou non, lished or not. The documents may come from émanant des établissements d’enseignement et de teaching and research institutions in France or recherche français ou étrangers, des laboratoires abroad, or from public or private research centers. publics ou privés. THÈSE POUR OBTENIR LE GRADE DE DOCTEUR DE L’UNIVERSITÉ DE MONTPELLIER En Biostatistique École doctorale I2S - Information, Structures, Systèmes Unité de recherche UMR 5149 - IMAG - Institut Montpelliérain Alexander Grothendieck Inférence statistique bayésienne pour les modélisations donnant lieu à un calcul de vraisemblance impossible Présentée par Louis RAYNAL Le 10 septembre 2019 Sous la direction de Jean-Michel MARIN Devant le jury composé de Mark BEAUMONT Professeur University of Bristol Rapporteur Michael BLUM Directeur de recherche Université de Grenoble Alpes Rapporteur Alice CLEYNEN Chargée de recherche Université de Montpellier Examinatrice
    [Show full text]
  • Bayesian Network Classifiers 133
    Machine Learning, 29, 131–163 (1997) c 1997 Kluwer Academic Publishers. Manufactured in The Netherlands. Bayesian Network Classifiers* NIR FRIEDMAN [email protected] Computer Science Division, 387 Soda Hall, University of California, Berkeley, CA 94720 DAN GEIGER [email protected] Computer Science Department, Technion, Haifa, Israel, 32000 MOISES GOLDSZMIDT [email protected] SRI International, 333 Ravenswood Ave., Menlo Park, CA 94025 Editor: G. Provan, P. Langley, and P. Smyth Abstract. Recent work in supervised learning has shown that a surprisingly simple Bayesian classifier with strong assumptions of independence among features, called naive Bayes, is competitive with state-of-the-art classifiers such as C4.5. This fact raises the question of whether a classifier with less restrictive assumptions can perform even better. In this paper we evaluate approaches for inducing classifiers from data, based on the theory of learning Bayesian networks. These networks are factored representations of probability distributions that generalize the naive Bayesian classifier and explicitly represent statements about independence. Among these approaches we single out a method we call Tree Augmented Naive Bayes (TAN), which outperforms naive Bayes, yet at the same time maintains the computational simplicity (no search involved) and robustness that characterize naive Bayes. We experimentally tested these approaches, using problems from the University of California at Irvine repository, and compared them to C4.5, naive Bayes, and wrapper methods for feature selection. Keywords: Bayesian networks, classification 1. Introduction Classification is a basic task in data analysis and pattern recognition that requires the construction of a classifier, that is, a function that assigns a class label to instances described by a set of attributes.
    [Show full text]
  • Course Notes for Bayesian Models for Machine Learning
    Course Notes for Bayesian Models for Machine Learning John Paisley Department of Electrical Engineering Columbia University Fall 2016 1 Contents Contents2 Probability review, Bayes rule, conjugate priors3 Bayesian linear regression, Bayes classifiers, predictive distributions 10 Laplace approximation, Gibbs sampling, logistic regression, matrix factorization 22 EM algorithm, probit regression 32 EM to variational inference 45 Variational inference, finding optimal distributions 57 Latent Dirichlet allocation, exponential families 67 conjugate exponential family models, scalable inference 77 Gaussian mixture models 87 Bayesian nonparametric clustering 98 Hidden Markov models 108 Poisson matrix factorization 118 2 EECS E6720 Bayesian Models for Machine Learning Columbia University, Fall 2016 Lecture 1, 9/8/2016 Instructor: John Paisley “Bayes rule” pops out of basic manipulations of probability distributions. Let’s reach it through • a very simple example. Example Call this entire space Ω Ai is the ith column (defined arbitrarily) Bi is the ith row (also defined arbitrarily) We have points lying in this space. We pick one of these points uniformly at random. • In this case, calculating probabilities is simply a matter of counting. • #A #B P (x A ) = 1 ,P (x B ) = 1 ∈ 1 #Ω ∈ 1 #Ω What about P (x A x B )? This is the probability that x A given that I know x B . • ∈ 1| ∈ 1 ∈ 1 ∈ 1 This is called a conditional probability. #(A B ) #(A B ) #Ω P (x A & x B ) P (x A x B ) = 1 ∩ 1 = 1 ∩ 1 = ∈ 1 ∈ 1 ∈ 1| ∈ 1 #B #Ω #B P (x B ) 1 1 ∈ 1 1 We’ve simply multiplied and divided by the same thing, but already we’ve made a general state- • ment.
    [Show full text]
  • Building Classifiers Using Bayesian Networks
    From: AAAI-96 Proceedings. Copyright © 1996, AAAI (www.aaai.org). All rights reserved. Building Classifiers using ayesian Networks Nir Friedman Moises Goldszmidt* Stanford University Rockwell Science Center Dept. of Computer Science 444 High St., Suite 400 Gates Building 1A Palo Alto, CA 94301 Stanford, CA 943059010 [email protected] [email protected] Abstract instantiation of Al, . , A,. This computation is rendered feasible by making a strong independence assumption: all Recent work in supervised learning has shown that a surpris- the attributes Ai are conditionally independent given the ingly simple Bayesian classifier with strong assumptions of value of the label C.’ The performance of naive Bayes is independence among features, called naive Bayes, is com- somewhat surprising given that this is clearly an unrealistic petitive with state of the art classifiers such as C4.5. This fact raises the question of whether a classifier with less re- assumption. Consider for example a classifier for assessing strictive assumptions can perform even better. In this paper the risk in loan applications. It would be erroneous to ignore we examine and evaluate approaches for inducing classifiers the correlations between age, education level, and income. from data, based on recent results in the theory of learn- This fact naturally begs the question of whether we can ing Bayesian networks. Bayesian networks are factored improve the performance of Bayesian classifiers by avoid- representations of probability distributions that generalize ing unrealistic assumptions about independence. In order the naive Bayes classifier and explicitly represent statements to effectively tackle this problem we need an appropriate about independence.
    [Show full text]
  • Pattern Recognition Chapter 4: Bayes Classifier
    Pattern Recognition Chapter 4: Bayes Classifier Asst. Prof. Dr. Chumphol Bunkhumpornpat Department of Computer Science Faculty of Science Chiang Mai University Learning Objectives • Be able to state the Bayes theorem • Be able to classify objects using naive Bayes classifiers • Be able to understand how Bayesian belief network 204453: Pattern Recognition 2 Bayes Classifier • Simple Probabilistic Classifier • Optimal Classifier – It minimises the average probability of error. • Assumption – Information about classes is in the form of prior probabilities. – Distributions of patterns in the class are known. 204453: Pattern Recognition 3 Bayes Classifier (cont.) • It employs the posterior probabilities to assign the class label to a test pattern. • A pattern is assigned the label of the class that has the maximum posterior probability. • Bayes theorem converts the prior probability into posterior probability based on the pattern to be classified. 204453: Pattern Recognition 4 Bayes Theorem • X: Pattern whose class label is unknown • Hi: Hypothesis that X belongs to class Ci • P(Hi): Prior probability of Hi – Known – Obtained before observing X • P(Hi |X): Posterior probability of Hi conditioned on X • P(Hi |X) = P(X|Hi) P(Hi) / P(X) 204453: Pattern Recognition 5 Example 1 • In a coffee shop, 99% of the customers prefer coffee. • The remaining 1% prefer tea. • P(coffee drinker) = 0.99 • P(tea drinker) = 0.01 204453: Pattern Recognition 6 Example 1 (cont.) • In the absence of any other information, we can classify any customer as a coffee drinker and the probability of error is only 0.01. • This is because we are classifying a tea drinker also as a coffee drinker.
    [Show full text]
  • Bayesian Inference in High-Dimensional Models
    Bayesian inference in high-dimensional models Sayantan Banerjee, Ismael Castillo and Subhashis Ghosal Abstract Models with dimension often more than the available sample size are now commonly used in various applications. A sensible inference is possible us- ing a lower-dimensional structure. In regression problems with a large number of predictors, the model is often assumed to be sparse, with only a few predictors ac- tive. Interdependence between a large number of variables is succinctly described by a graphical model, where variables are represented by nodes on a graph and an edge between two nodes is used to indicate their conditional dependence given other variables. Many procedures for making inferences in the high-dimensional setting, typically using penalty functions to induce sparsity in the solution obtained by min- imizing a loss function, were developed. Bayesian methods have been proposed for such problems more recently, where the prior takes care of the sparsity structure. These methods have the natural ability to also automatically quantify the uncertainty of the inference through the posterior distribution. Theoretical studies of Bayesian procedures in high-dimension have been carried out recently. Questions that arise are whether the posterior distribution contracts near the true value of the param- eter at the minimax optimal rate, whether the correct lower-dimensional structure is discovered with high posterior probability, and if a credible region has adequate frequentist coverage. In this paper, we review the properties of Bayesian and related methods for several high-dimensional models such as many normal means problem, linear regression, generalized linear models, Gaussian and non-Gaussian graphical models.
    [Show full text]